Max level shown:
1 [ Aug 14 22:27:32 Enabled. ]
2 [ Aug 14 22:27:32 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Aug 14 22:27:32 Method "start" exited with status 0. ]
4 [ Aug 14 22:27:32 Rereading configuration. ]
5 [ Aug 14 22:27:32 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-08-14T22:27:32.265ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-08-14T22:27:32.265ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-08-14T22:27:32.317ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-08-14T22:27:32.317ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-08-14T22:27:32.433ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-08-14T22:27:32.442ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-08-14T22:27:32.450ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-08-14T22:27:33.063ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-08-14T22:27:33.201ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:125
162024-08-14T22:27:33.201ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:137
172024-08-14T22:27:33.201ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:150
182024-08-14T22:27:33.201ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:172 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-08-14T22:27:33.201ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
202024-08-14T22:27:33.202ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
212024-08-14T22:27:33.204ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-08-14T22:27:33.204ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-08-14T22:27:33.204ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:186
242024-08-14T22:27:33.204ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-08-14T22:27:33.204ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_0.vdev
262024-08-14T22:27:33.204ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-08-14T22:27:33.205ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
282024-08-14T22:27:33.205ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-08-14T22:27:33.205ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-08-14T22:27:33.216ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-08-14T22:27:33.216ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = eb7d1608-061f-4633-a5da-2133d1a63302 (zpool)
322024-08-14T22:27:33.316ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal }
332024-08-14T22:27:33.522ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:405
342024-08-14T22:27:33.522ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_1.vdev
352024-08-14T22:27:33.522ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-08-14T22:27:33.522ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-08-14T22:27:33.522ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
382024-08-14T22:27:33.522ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-08-14T22:27:33.522ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-08-14T22:27:33.531ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-08-14T22:27:33.531ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool)
422024-08-14T22:27:33.617ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal }
432024-08-14T22:27:33.815ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_0.vdev
442024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
472024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
482024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_1.vdev
502024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
522024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_2.vdev
562024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
582024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
592024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
602024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_3.vdev
622024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
662024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
682024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
692024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
702024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_4.vdev
712024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
722024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_5.vdev
742024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
782024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_6.vdev
802024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
812024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
822024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
832024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_7.vdev
862024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
872024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-08-14T22:27:33.816ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_8.vdev
922024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
932024-08-14T22:27:33.816ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-08-14T22:27:33.816ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:96
982024-08-14T22:27:33.816ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:98
992024-08-14T22:27:33.816ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1002024-08-14T22:27:33.816ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1012024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1022024-08-14T22:27:33.816ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-08-14T22:27:33.816ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:210
1042024-08-14T22:27:33.816ZINFOSledAgent (bootstore): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-08-14T22:27:33.816ZINFOSledAgent (bootstore): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-08-14T22:27:33.816ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-08-14T22:27:33.816ZINFOSledAgent (bootstore): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-08-14T22:27:33.816ZINFOSledAgent (bootstore): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-08-14T22:27:33.816ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-08-14T22:27:33.816ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:215
1112024-08-14T22:27:33.816ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:230
1122024-08-14T22:27:33.827ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:730
1132024-08-14T22:27:33.827ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 11866, tv_nsec: 635662504 } time_to_next_cleanup = 599.999939069s
1142024-08-14T22:27:33.827ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3778
1152024-08-14T22:27:33.827ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1162024-08-14T22:27:33.827ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-08-14T22:27:33.830ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-08-14T22:27:33.830ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-08-14T22:27:33.830ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-08-14T22:27:33.831ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1017
1212024-08-14T22:27:33.831ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-08-14T22:27:33.831ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1232024-08-14T22:27:33.831ZINFOSledAgent (RSS): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-08-14T22:27:33.831ZINFOSledAgent (RSS): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-08-14T22:27:33.831ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1077
1262024-08-14T22:27:33.831ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1272024-08-14T22:27:33.831ZINFOSledAgent (RSS): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-08-14T22:27:33.831ZINFOSledAgent (RSS): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-08-14T22:27:33.831ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1118
1302024-08-14T22:27:33.831ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-08-14T22:27:33.831ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-08-14T22:27:33.831ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1332024-08-14T22:27:33.831ZINFOSledAgent (RSS): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-08-14T22:27:33.831ZINFOSledAgent (RSS): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-08-14T22:27:33.831ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-08-14T22:27:33.831ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1151
1372024-08-14T22:27:33.831ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-08-14T22:27:33.831ZINFOSledAgent (bootstore): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-08-14T22:27:33.832ZINFOSledAgent (bootstore): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-08-14T22:27:33.832ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-08-14T22:27:33.832ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: aeb7e077-0751-417b-8b16-b62876b5611c, rack_id: a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-08-14T22:27:33.832ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:46113
1432024-08-14T22:27:33.832ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: aeb7e077-0751-417b-8b16-b62876b5611c, rack_id: a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-08-14T22:27:33.832ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-08-14T22:27:33.832ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-08-14T22:27:33.832ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1472024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:405
1482024-08-14T22:27:33.832ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1492024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:573
1502024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1512024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:562
1532024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:590
1542024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:394
1552024-08-14T22:27:33.832ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
1562024-08-14T22:27:33.832ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:480 result = DisksManagementResult { status: [] }
1572024-08-14T22:27:33.834ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4012
1582024-08-14T22:27:33.834ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4394
1592024-08-14T22:27:33.865ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1602024-08-14T22:27:33.886ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-08-14T22:27:33.904ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-08-14T22:27:33.949ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:382 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1632024-08-14T22:27:33.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1642024-08-14T22:27:33.949ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:394 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1652024-08-14T22:27:33.949ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-08-14T22:27:33.952ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 114.325436ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "22dc383e-35af-43de-a923-48d869856fbc", "content-length": "124", "date": "Wed, 14 Aug 2024 22:27:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "22dc383e-35af-43de-a923-48d869856fbc" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-08-14T22:27:33.963ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_eb7d1608-061f-4633-a5da-2133d1a63302/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-08-14T22:27:33.988ZINFOSledAgent: successfully created encrypted zvol: name="oxi_eb7d1608-061f-4633-a5da-2133d1a63302/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-08-14T22:27:33.988ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_eb7d1608-061f-4633-a5da-2133d1a63302/swap" file = sled-agent/src/swap_device.rs:96
1702024-08-14T22:27:33.994ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:409 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1712024-08-14T22:27:33.994ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1722024-08-14T22:27:33.994ZINFOSledAgent (BackingFs): Ensuring dataset oxi_eb7d1608-061f-4633-a5da-2133d1a63302/backing/fmd file = sled-agent/src/backing_fs.rs:136
1732024-08-14T22:27:34.068ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-08-14T22:27:34.068ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 162.131639ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f2440292-f776-466a-b324-5a206dd1567a", "content-length": "124", "date": "Wed, 14 Aug 2024 22:27:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f2440292-f776-466a-b324-5a206dd1567a" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-08-14T22:27:34.072ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1762024-08-14T22:27:34.232ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-08-14T22:27:34.233ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 206.746301ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a27cfcca-9a9c-4ac5-8efb-f8431adf0c12", "content-length": "124", "date": "Wed, 14 Aug 2024 22:27:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a27cfcca-9a9c-4ac5-8efb-f8431adf0c12" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-08-14T22:27:34.291ZINFOSledAgent (BackingFs): Mounting oxi_eb7d1608-061f-4633-a5da-2133d1a63302/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1792024-08-14T22:27:34.303ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1802024-08-14T22:27:34.304ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1812024-08-14T22:27:34.304ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1822024-08-14T22:27:34.304ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1832024-08-14T22:27:34.441ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-08-14T22:27:34.442ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 183.959679ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f7a8b258-edc5-473c-90f9-1835dd540aae", "content-length": "124", "date": "Wed, 14 Aug 2024 22:27:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f7a8b258-edc5-473c-90f9-1835dd540aae" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-08-14T22:27:34.627ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-08-14T22:27:34.627ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 130.882848ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d23a9240-815e-4cfe-841e-37930083b66f", "content-length": "124", "date": "Wed, 14 Aug 2024 22:27:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d23a9240-815e-4cfe-841e-37930083b66f" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-08-14T22:27:34.759ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-08-14T22:27:34.760ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 362.166237ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "79c8a343-639d-4c1d-a6b0-230cae08cbd1", "content-length": "124", "date": "Wed, 14 Aug 2024 22:27:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "79c8a343-639d-4c1d-a6b0-230cae08cbd1" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-08-14T22:27:35.039ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1902024-08-14T22:27:35.044ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:49293 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1912024-08-14T22:27:35.044ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1922024-08-14T22:27:35.044ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:49293 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = aeb7e077-0751-417b-8b16-b62876b5611c sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1932024-08-14T22:27:35.046ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 215.248924ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1942024-08-14T22:27:35.047ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1952024-08-14T22:27:35.047ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
1962024-08-14T22:27:35.047ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:914 underlay_address = fd00:1122:3344:101::1
1972024-08-14T22:27:35.047ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1982024-08-14T22:27:35.123ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1992024-08-14T22:27:35.262ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 299.836041ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2002024-08-14T22:27:35.563ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 500.997507ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2012024-08-14T22:27:36.066ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.122089935s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2022024-08-14T22:27:37.122ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2551
2032024-08-14T22:27:37.123ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2769
2042024-08-14T22:27:37.123ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2421
2052024-08-14T22:27:37.123ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2468
2062024-08-14T22:27:37.123ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2850
2072024-08-14T22:27:37.123ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2892
2082024-08-14T22:27:37.125ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2694
2092024-08-14T22:27:37.126ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2462
2102024-08-14T22:27:37.126ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='52a18bc2-4a0b-4245-b6ae-36905229844c'/> <propval type="astring" name="rack_id" value='a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fec8:642d'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='aeb7e077-0751-417b-8b16-b62876b5611c'/> <propval type="astring" name="rack_id" value='a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='aeb7e077-0751-417b-8b16-b62876b5611c'/> <propval type="astring" name="rack_uuid" value='a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='aeb7e077-0751-417b-8b16-b62876b5611c'/> <propval type="astring" name="rack_uuid" value='a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2112024-08-14T22:27:37.126ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2122024-08-14T22:27:37.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 51.766554ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2132024-08-14T22:27:37.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 94.63548ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2142024-08-14T22:27:37.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 254.684013ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2152024-08-14T22:27:38.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 250.228382ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2162024-08-14T22:27:38.189ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 5.75115861s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2172024-08-14T22:27:38.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 643.977573ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2182024-08-14T22:27:39.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 673.758485ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-08-14T22:27:40.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.363188042s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-08-14T22:27:41.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 716.559407ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-08-14T22:27:42.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.177060305s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-08-14T22:27:44.310ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.914605313s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2232024-08-14T22:27:44.316ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2242024-08-14T22:27:44.316ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2252024-08-14T22:27:44.316ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2262024-08-14T22:27:44.316ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2272024-08-14T22:27:44.316ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2282024-08-14T22:27:44.316ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2292024-08-14T22:27:44.316ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2302024-08-14T22:27:44.316ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3778
2312024-08-14T22:27:44.316ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:46113
2322024-08-14T22:27:44.540ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2332024-08-14T22:27:44.540ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2342024-08-14T22:27:44.540ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2352024-08-14T22:27:44.540ZINFOSledAgent (RSS): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2362024-08-14T22:27:44.540ZINFOSledAgent (RSS): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2372024-08-14T22:27:44.540ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2382024-08-14T22:27:44.540ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2392024-08-14T22:27:44.545ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4036 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2402024-08-14T22:27:44.545ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4052
2412024-08-14T22:27:44.545ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2422024-08-14T22:27:44.815ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65092
2432024-08-14T22:27:44.815ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2442024-08-14T22:27:44.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 302 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65092 req_id = a369e14d-ae64-4d22-b3a4-1fd4279dd587 response_code = 200 uri = /inventory
2452024-08-14T22:27:44.846ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36444
2462024-08-14T22:27:44.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36444 req_id = d0ac661b-9087-4477-bc1a-8e72c7861cdb response_code = 200 uri = /sled-role
2472024-08-14T22:27:44.847ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2482024-08-14T22:27:44.847ZINFOSledAgent (RSS): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2492024-08-14T22:27:44.848ZINFOSledAgent (RSS): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2502024-08-14T22:27:44.848ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:828
2512024-08-14T22:27:44.880ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: b21fff50-4c27-4f69-85a9-2e9d0136759e, pool_id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 42cf2721-b923-4b2e-817d-6adff83fb442, pool_id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: ec31ea05-7f2c-4cbd-bd06-b4ccd1ee7a39, pool_id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: d1067799-d609-4b88-9cad-128be7f8d5bf, pool_id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 7049b87c-122c-4d5a-9d83-954a9fc4e6b0, pool_id: 20226c3a-235f-4118-b164-e14967848f5c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: a889f415-e9cf-430e-a36b-21ed285d732b, pool_id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 04103578-ce32-4b57-bfa9-17fa3ec0a518, pool_id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 629b415d-04e1-4664-acbf-66f4915c5271, pool_id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: af847740-5ca2-41b6-8c05-34a1bb9a547a, pool_id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:355 sled_address = [fd00:1122:3344:101::1]:12345
2522024-08-14T22:27:44.880ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60763
2532024-08-14T22:27:44.880ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:831 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2542024-08-14T22:27:44.880ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: b21fff50-4c27-4f69-85a9-2e9d0136759e, pool_id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 42cf2721-b923-4b2e-817d-6adff83fb442, pool_id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: ec31ea05-7f2c-4cbd-bd06-b4ccd1ee7a39, pool_id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: d1067799-d609-4b88-9cad-128be7f8d5bf, pool_id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 7049b87c-122c-4d5a-9d83-954a9fc4e6b0, pool_id: 20226c3a-235f-4118-b164-e14967848f5c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: a889f415-e9cf-430e-a36b-21ed285d732b, pool_id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 04103578-ce32-4b57-bfa9-17fa3ec0a518, pool_id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 629b415d-04e1-4664-acbf-66f4915c5271, pool_id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: af847740-5ca2-41b6-8c05-34a1bb9a547a, pool_id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:405
2552024-08-14T22:27:44.880ZINFOSledAgent (StorageManager): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2562024-08-14T22:27:44.880ZINFOSledAgent (StorageManager): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2572024-08-14T22:27:44.880ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:644 request = omicron_physical_disks_ensure
2582024-08-14T22:27:44.880ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
2592024-08-14T22:27:44.880ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:414
2602024-08-14T22:27:44.880ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2612024-08-14T22:27:44.880ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2622024-08-14T22:27:45.303ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2632024-08-14T22:27:45.305ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2642024-08-14T22:27:45.305ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 17668df1-993b-458e-b1cf-310a88477d11 (zpool)
2652024-08-14T22:27:45.317ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
2662024-08-14T22:27:45.402ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }
2672024-08-14T22:27:45.410ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:348
2682024-08-14T22:27:45.410ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:350
2692024-08-14T22:27:45.410ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:354
2702024-08-14T22:27:45.410ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:356
2712024-08-14T22:27:45.410ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2722024-08-14T22:27:45.411ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2732024-08-14T22:27:45.411ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
2742024-08-14T22:27:45.445ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2752024-08-14T22:27:45.453ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_17668df1-993b-458e-b1cf-310a88477d11/crypt/zone file = sled-storage/src/dataset.rs:411
2762024-08-14T22:27:45.576ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_17668df1-993b-458e-b1cf-310a88477d11 file = sled-storage/src/dataset.rs:496
2772024-08-14T22:27:45.584ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:512
2782024-08-14T22:27:45.584ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:429
2792024-08-14T22:27:45.584ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:414
2802024-08-14T22:27:45.584ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2812024-08-14T22:27:45.584ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2822024-08-14T22:27:45.593ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2832024-08-14T22:27:45.593ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool)
2842024-08-14T22:27:45.688ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }
2852024-08-14T22:27:45.696ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:348
2862024-08-14T22:27:45.696ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:350
2872024-08-14T22:27:45.696ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:354
2882024-08-14T22:27:45.696ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:356
2892024-08-14T22:27:45.696ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2902024-08-14T22:27:45.696ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2912024-08-14T22:27:45.696ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
2922024-08-14T22:27:45.730ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2932024-08-14T22:27:45.737ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7f6d9db3-762e-4798-98cd-bb00d0a3b252/crypt/zone file = sled-storage/src/dataset.rs:411
2942024-08-14T22:27:45.858ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7f6d9db3-762e-4798-98cd-bb00d0a3b252 file = sled-storage/src/dataset.rs:496
2952024-08-14T22:27:45.866ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:512
2962024-08-14T22:27:45.866ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:429
2972024-08-14T22:27:45.866ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:414
2982024-08-14T22:27:45.866ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2992024-08-14T22:27:45.866ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3002024-08-14T22:27:45.875ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3012024-08-14T22:27:45.875ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool)
3022024-08-14T22:27:45.967ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }
3032024-08-14T22:27:45.975ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:348
3042024-08-14T22:27:45.975ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:350
3052024-08-14T22:27:45.975ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:354
3062024-08-14T22:27:45.975ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:356
3072024-08-14T22:27:45.975ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3082024-08-14T22:27:45.975ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3092024-08-14T22:27:45.975ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3102024-08-14T22:27:46.012ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3112024-08-14T22:27:46.020ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1bd3ff04-304c-4362-b41f-8347b5dd9341/crypt/zone file = sled-storage/src/dataset.rs:411
3122024-08-14T22:27:46.141ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1bd3ff04-304c-4362-b41f-8347b5dd9341 file = sled-storage/src/dataset.rs:496
3132024-08-14T22:27:46.149ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:512
3142024-08-14T22:27:46.149ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:429
3152024-08-14T22:27:46.149ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:414
3162024-08-14T22:27:46.149ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3172024-08-14T22:27:46.149ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3182024-08-14T22:27:46.158ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3192024-08-14T22:27:46.158ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool)
3202024-08-14T22:27:46.249ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }
3212024-08-14T22:27:46.257ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:348
3222024-08-14T22:27:46.257ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:350
3232024-08-14T22:27:46.257ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:354
3242024-08-14T22:27:46.257ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:356
3252024-08-14T22:27:46.257ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3262024-08-14T22:27:46.257ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3272024-08-14T22:27:46.257ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3282024-08-14T22:27:46.293ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3292024-08-14T22:27:46.300ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_db606de8-ae75-4597-bde9-3f1234d88ff3/crypt/zone file = sled-storage/src/dataset.rs:411
3302024-08-14T22:27:46.319ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
3312024-08-14T22:27:46.415ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_db606de8-ae75-4597-bde9-3f1234d88ff3 file = sled-storage/src/dataset.rs:496
3322024-08-14T22:27:46.423ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:512
3332024-08-14T22:27:46.423ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:429
3342024-08-14T22:27:46.423ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:414
3352024-08-14T22:27:46.423ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3362024-08-14T22:27:46.423ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3372024-08-14T22:27:46.433ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3382024-08-14T22:27:46.433ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 20226c3a-235f-4118-b164-e14967848f5c (zpool)
3392024-08-14T22:27:46.527ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }
3402024-08-14T22:27:46.535ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:348
3412024-08-14T22:27:46.535ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:350
3422024-08-14T22:27:46.535ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:354
3432024-08-14T22:27:46.535ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:356
3442024-08-14T22:27:46.535ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3452024-08-14T22:27:46.535ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3462024-08-14T22:27:46.535ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3472024-08-14T22:27:46.569ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3482024-08-14T22:27:46.577ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_20226c3a-235f-4118-b164-e14967848f5c/crypt/zone file = sled-storage/src/dataset.rs:411
3492024-08-14T22:27:46.702ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_20226c3a-235f-4118-b164-e14967848f5c file = sled-storage/src/dataset.rs:496
3502024-08-14T22:27:46.710ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:512
3512024-08-14T22:27:46.710ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:429
3522024-08-14T22:27:46.710ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:414
3532024-08-14T22:27:46.710ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3542024-08-14T22:27:46.710ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3552024-08-14T22:27:46.719ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3562024-08-14T22:27:46.719ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool)
3572024-08-14T22:27:46.813ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool), kind: External }
3582024-08-14T22:27:46.821ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:348
3592024-08-14T22:27:46.821ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:350
3602024-08-14T22:27:46.821ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:354
3612024-08-14T22:27:46.821ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:356
3622024-08-14T22:27:46.821ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3632024-08-14T22:27:46.821ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3642024-08-14T22:27:46.821ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3652024-08-14T22:27:46.857ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3662024-08-14T22:27:46.865ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2450844a-e7c2-49d0-8dc6-d59793fa3c37/crypt/zone file = sled-storage/src/dataset.rs:411
3672024-08-14T22:27:46.995ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2450844a-e7c2-49d0-8dc6-d59793fa3c37 file = sled-storage/src/dataset.rs:496
3682024-08-14T22:27:47.003ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:512
3692024-08-14T22:27:47.003ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:429
3702024-08-14T22:27:47.003ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:414
3712024-08-14T22:27:47.003ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3722024-08-14T22:27:47.003ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3732024-08-14T22:27:47.012ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3742024-08-14T22:27:47.012ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool)
3752024-08-14T22:27:47.111ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }
3762024-08-14T22:27:47.119ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:348
3772024-08-14T22:27:47.119ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:350
3782024-08-14T22:27:47.119ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:354
3792024-08-14T22:27:47.119ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:356
3802024-08-14T22:27:47.119ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3812024-08-14T22:27:47.120ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3822024-08-14T22:27:47.120ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3832024-08-14T22:27:47.155ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3842024-08-14T22:27:47.164ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a4af3495-6c0f-4fe3-b8a0-503d2a880027/crypt/zone file = sled-storage/src/dataset.rs:411
3852024-08-14T22:27:47.287ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a4af3495-6c0f-4fe3-b8a0-503d2a880027 file = sled-storage/src/dataset.rs:496
3862024-08-14T22:27:47.296ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:512
3872024-08-14T22:27:47.296ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:429
3882024-08-14T22:27:47.296ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:414
3892024-08-14T22:27:47.296ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3902024-08-14T22:27:47.296ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3912024-08-14T22:27:47.306ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3922024-08-14T22:27:47.306ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool)
3932024-08-14T22:27:47.402ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool), kind: External }
3942024-08-14T22:27:47.410ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:348
3952024-08-14T22:27:47.410ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:350
3962024-08-14T22:27:47.410ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:354
3972024-08-14T22:27:47.410ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:356
3982024-08-14T22:27:47.410ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3992024-08-14T22:27:47.410ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4002024-08-14T22:27:47.410ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
4012024-08-14T22:27:47.414ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4022024-08-14T22:27:47.447ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4032024-08-14T22:27:47.454ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_0faf5893-5a63-4906-a7f5-cb36c6c1353b/crypt/zone file = sled-storage/src/dataset.rs:411
4042024-08-14T22:27:47.576ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_0faf5893-5a63-4906-a7f5-cb36c6c1353b file = sled-storage/src/dataset.rs:496
4052024-08-14T22:27:47.583ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:512
4062024-08-14T22:27:47.583ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:429
4072024-08-14T22:27:47.583ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:414
4082024-08-14T22:27:47.583ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
4092024-08-14T22:27:47.583ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4102024-08-14T22:27:47.593ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4112024-08-14T22:27:47.593ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool)
4122024-08-14T22:27:47.688ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool), kind: External }
4132024-08-14T22:27:47.696ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:348
4142024-08-14T22:27:47.696ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:350
4152024-08-14T22:27:47.696ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:354
4162024-08-14T22:27:47.696ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:356
4172024-08-14T22:27:47.696ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4182024-08-14T22:27:47.696ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4192024-08-14T22:27:47.696ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
4202024-08-14T22:27:47.731ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4212024-08-14T22:27:47.739ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_126123c3-ef9f-486e-8f31-7272296cc4d8/crypt/zone file = sled-storage/src/dataset.rs:411
4222024-08-14T22:27:47.864ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_126123c3-ef9f-486e-8f31-7272296cc4d8 file = sled-storage/src/dataset.rs:496
4232024-08-14T22:27:47.873ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:512
4242024-08-14T22:27:47.873ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:429
4252024-08-14T22:27:47.873ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4262024-08-14T22:27:47.873ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: eb7d1608-061f-4633-a5da-2133d1a63302 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2e8af2e9-8342-42ed-88bb-bcb6bcd354d3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4272024-08-14T22:27:47.873ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4282024-08-14T22:27:47.873ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4292024-08-14T22:27:47.873ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4302024-08-14T22:27:47.873ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:835 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4312024-08-14T22:27:47.873ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4322024-08-14T22:27:47.873ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:861 generation = Generation(1) sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4332024-08-14T22:27:47.873ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:866 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4342024-08-14T22:27:47.873ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:871 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4352024-08-14T22:27:47.874ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:120 gen = Generation(1) sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4362024-08-14T22:27:47.874ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:877 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4372024-08-14T22:27:47.874ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:803 gen = Generation(1)
4382024-08-14T22:27:47.874ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:882 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4392024-08-14T22:27:47.874ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 2993489 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:60763 req_id = a53ecbf4-68e2-4c19-8c8a-d8e5c715bbcc response_code = 200 uri = /omicron-physical-disks
4402024-08-14T22:27:47.905ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
4412024-08-14T22:27:47.905ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50908
4422024-08-14T22:27:47.905ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4432024-08-14T22:27:47.905ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4442024-08-14T22:27:47.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4452024-08-14T22:27:47.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4462024-08-14T22:27:48.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4472024-08-14T22:27:48.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4482024-08-14T22:27:48.057ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4492024-08-14T22:27:48.057ZINFOSledAgent (ServiceManager): No ledger in /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:188
4502024-08-14T22:27:48.057ZINFOSledAgent (ServiceManager): No ledger in /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:188
4512024-08-14T22:27:48.057ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4522024-08-14T22:27:48.101ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4532024-08-14T22:27:48.101ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4542024-08-14T22:27:48.101ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4552024-08-14T22:27:48.154ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0 file = illumos-utils/src/zone.rs:316
4562024-08-14T22:27:48.158ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d file = illumos-utils/src/zone.rs:316
4572024-08-14T22:27:48.162ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53 file = illumos-utils/src/zone.rs:316
4582024-08-14T22:27:48.175ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0 file = illumos-utils/src/zone.rs:354
4592024-08-14T22:27:48.179ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d file = illumos-utils/src/zone.rs:354
4602024-08-14T22:27:48.182ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53 file = illumos-utils/src/zone.rs:354
4612024-08-14T22:27:48.417ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
4622024-08-14T22:27:49.686ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4632024-08-14T22:27:49.686ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4642024-08-14T22:27:49.686ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4652024-08-14T22:27:50.171ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4662024-08-14T22:27:50.171ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4672024-08-14T22:27:50.651ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4682024-08-14T22:27:50.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 42.413819ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4692024-08-14T22:27:50.658ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4702024-08-14T22:27:50.661ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4712024-08-14T22:27:50.661ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4722024-08-14T22:27:50.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 131.405494ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4732024-08-14T22:27:50.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 31.305288ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4742024-08-14T22:27:50.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 33.98682ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4752024-08-14T22:27:50.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 75.701248ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4762024-08-14T22:27:50.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 93.14138ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4772024-08-14T22:27:50.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 235.138221ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4782024-08-14T22:27:51.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 115.379559ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4792024-08-14T22:27:51.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 161.225785ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4802024-08-14T22:27:51.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 284.45604ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4812024-08-14T22:27:51.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 377.245462ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4822024-08-14T22:27:51.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 527.028233ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4832024-08-14T22:27:51.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 886.012238ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4842024-08-14T22:27:51.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 620.962973ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4852024-08-14T22:27:51.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 784.614987ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4862024-08-14T22:27:52.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 939.418466ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4872024-08-14T22:27:52.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 1.369863457s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4882024-08-14T22:27:52.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 747.860615ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4892024-08-14T22:27:53.239ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 661.996495ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
4902024-08-14T22:27:53.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 1.364641084s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
4912024-08-14T22:27:53.632ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4060
4922024-08-14T22:27:53.632ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4070 gateway = fd00:1122:3344:101::1
4932024-08-14T22:27:53.638ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4149
4942024-08-14T22:27:53.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 1.423789269s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
4952024-08-14T22:27:53.764ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4197
4962024-08-14T22:27:53.764ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4224
4972024-08-14T22:27:53.803ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4237
4982024-08-14T22:27:53.803ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4101
4992024-08-14T22:27:53.856ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4143
5002024-08-14T22:27:53.857ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4204 rack_subnet = fd00:1122:3344::
5012024-08-14T22:27:53.876ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4215
5022024-08-14T22:27:53.876ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4257
5032024-08-14T22:27:53.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 916.343146ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
5042024-08-14T22:27:53.960ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4289
5052024-08-14T22:27:53.960ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4295
5062024-08-14T22:27:54.070ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4336
5072024-08-14T22:27:54.070ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5082024-08-14T22:27:54.102ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
5092024-08-14T22:27:54.102ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 58.355728ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5102024-08-14T22:27:54.162ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 114.78382ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5112024-08-14T22:27:54.277ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 292.026353ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5122024-08-14T22:27:54.571ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 472.21095ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5132024-08-14T22:27:54.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:41326
5142024-08-14T22:27:54.719ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 567 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:41326 req_id = c53c3ad8-9410-4f86-9892-93bbb1dc608e response_code = 200 uri = /sled-identifiers
5152024-08-14T22:27:54.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 564.508692ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
5162024-08-14T22:27:54.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 606.881819ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
5172024-08-14T22:27:55.045ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5182024-08-14T22:27:55.077ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:675
5192024-08-14T22:27:55.077ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:713
5202024-08-14T22:27:55.077ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:716 uptime = 1
5212024-08-14T22:27:55.077ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5222024-08-14T22:27:55.114ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
5232024-08-14T22:27:55.135ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
5242024-08-14T22:27:55.157ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:405
5252024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5262024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5272024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5282024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5292024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5302024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5312024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5322024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5332024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5342024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5352024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5362024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5372024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5382024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5392024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5402024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5412024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5422024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5432024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5442024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5452024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5462024-08-14T22:27:55.157ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5472024-08-14T22:27:55.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0") failed: Property not found. retry in 1.481976545s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
5482024-08-14T22:27:55.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53") failed: Property not found. retry in 756.497449ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
5492024-08-14T22:27:55.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d") failed: Property not found. retry in 998.823065ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
5502024-08-14T22:27:56.671ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_internal_dns_0895b1cd-15b7-45f6-983c-9e8b876b7c53
5512024-08-14T22:27:56.671ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_internal_dns_f6dfbfea-8bf4-4493-96d9-9d1b04d801d0
5522024-08-14T22:27:56.671ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_internal_dns_e319def5-0926-47c0-a571-e5e4b12c3c0d
5532024-08-14T22:27:56.671ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5542024-08-14T22:27:56.672ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5552024-08-14T22:27:56.672ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 8766997 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:50908 req_id = 67dddfe0-ce0f-4a91-8766-1d20c1f0998b response_code = 204 uri = /omicron-zones
5562024-08-14T22:27:56.674ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5572024-08-14T22:27:56.727ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5582024-08-14T22:27:56.774ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5592024-08-14T22:27:56.821ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:591
5602024-08-14T22:27:56.821ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5612024-08-14T22:27:56.851ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5622024-08-14T22:27:56.853ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5632024-08-14T22:27:56.853ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5642024-08-14T22:27:56.883ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
5652024-08-14T22:27:56.883ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42779
5662024-08-14T22:27:56.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5672024-08-14T22:27:56.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5682024-08-14T22:27:56.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5692024-08-14T22:27:56.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5702024-08-14T22:27:56.923ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5712024-08-14T22:27:56.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5722024-08-14T22:27:56.942ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5732024-08-14T22:27:56.942ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
5742024-08-14T22:27:56.942ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
5752024-08-14T22:27:56.943ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5762024-08-14T22:27:56.953ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5772024-08-14T22:27:56.986ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5782024-08-14T22:27:56.987ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5792024-08-14T22:27:56.987ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5802024-08-14T22:27:57.018ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 170, 161]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5812024-08-14T22:27:57.018ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = boundary_ntp
5822024-08-14T22:27:57.018ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5832024-08-14T22:27:57.018ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5842024-08-14T22:27:57.019ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5852024-08-14T22:27:57.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5862024-08-14T22:27:57.043ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_848ae27a-190f-4269-b82e-d54727102895 file = illumos-utils/src/zone.rs:316
5872024-08-14T22:27:57.061ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_848ae27a-190f-4269-b82e-d54727102895 file = illumos-utils/src/zone.rs:354
5882024-08-14T22:27:57.681ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_848ae27a-190f-4269-b82e-d54727102895: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5892024-08-14T22:27:57.681ZINFOSledAgent (ServiceManager): Booting oxz_ntp_848ae27a-190f-4269-b82e-d54727102895 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5902024-08-14T22:27:57.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 40.891118ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5912024-08-14T22:27:57.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 63.05547ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5922024-08-14T22:27:57.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 171.800253ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5932024-08-14T22:27:58.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 454.921706ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5942024-08-14T22:27:58.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 809.977979ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5952024-08-14T22:27:59.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 801.110671ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5962024-08-14T22:28:00.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 712.362311ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5972024-08-14T22:28:00.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 1.074738322s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
5982024-08-14T22:28:01.105ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: dns error: no record found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
5992024-08-14T22:28:02.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 1.018223508s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
6002024-08-14T22:28:03.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_848ae27a-190f-4269-b82e-d54727102895") failed: Property not found. retry in 1.452718233s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
6012024-08-14T22:28:04.523ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_ntp_848ae27a-190f-4269-b82e-d54727102895
6022024-08-14T22:28:04.523ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6032024-08-14T22:28:04.524ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6042024-08-14T22:28:04.524ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 7641443 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42779 req_id = cf71f498-ff9e-4569-a6f2-cabc9fe07844 response_code = 204 uri = /omicron-zones
6052024-08-14T22:28:04.526ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:632
6062024-08-14T22:28:04.556ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6072024-08-14T22:28:04.556ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60229
6082024-08-14T22:28:04.565ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_848ae27a-190f-4269-b82e-d54727102895': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3764
6092024-08-14T22:28:04.565ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 8972 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60229 req_id = d32aab04-8032-4107-8f86-6926ea4a102e response_code = 500 uri = /timesync
6102024-08-14T22:28:04.567ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6112024-08-14T22:28:04.692ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6122024-08-14T22:28:04.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42642
6132024-08-14T22:28:08.701ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_848ae27a-190f-4269-b82e-d54727102895': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3764
6142024-08-14T22:28:08.701ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 4008502 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42642 req_id = 65dd7530-cc40-4688-a81f-50a5e95615de response_code = 500 uri = /timesync
6152024-08-14T22:28:08.705ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6162024-08-14T22:28:08.810ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6172024-08-14T22:28:08.810ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55312
6182024-08-14T22:28:08.818ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_848ae27a-190f-4269-b82e-d54727102895': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3764
6192024-08-14T22:28:08.818ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 8451 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55312 req_id = 25b033b8-03e9-4318-9f6a-8169775fee2f response_code = 500 uri = /timesync
6202024-08-14T22:28:08.820ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6212024-08-14T22:28:08.972ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6222024-08-14T22:28:08.972ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46158
6232024-08-14T22:28:12.640ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_848ae27a-190f-4269-b82e-d54727102895': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3764
6242024-08-14T22:28:12.640ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 4009511 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46158 req_id = 7bda5881-4f67-474f-ad1d-59020bba9238 response_code = 500 uri = /timesync
6252024-08-14T22:28:12.642ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6262024-08-14T22:28:13.211ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6272024-08-14T22:28:13.212ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48635
6282024-08-14T22:28:13.220ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 8401 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48635 req_id = a72a7bac-1c18-4196-b49d-43356ef7bffc response_code = 200 uri = /timesync
6292024-08-14T22:28:13.223ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6302024-08-14T22:28:13.223ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6312024-08-14T22:28:13.413ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6322024-08-14T22:28:13.413ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49091
6332024-08-14T22:28:13.421ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 8239 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49091 req_id = 0112a93e-5080-4a5d-a41c-de40f2f079e1 response_code = 200 uri = /timesync
6342024-08-14T22:28:13.422ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6352024-08-14T22:28:13.422ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6362024-08-14T22:28:13.321ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6372024-08-14T22:28:13.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33458
6382024-08-14T22:28:13.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 20626 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33458 req_id = 4ea078fd-213d-4e3f-9637-108e276548fe response_code = 200 uri = /timesync
6392024-08-14T22:28:13.345ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2873254270, ip_addr: 171.66.97.126, stratum: 3, ref_time: 1723674493.199916, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6402024-08-14T22:28:13.345ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1230
6412024-08-14T22:28:13.376ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6422024-08-14T22:28:13.376ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45096
6432024-08-14T22:28:13.377ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6442024-08-14T22:28:13.377ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6452024-08-14T22:28:13.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6462024-08-14T22:28:13.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6472024-08-14T22:28:13.413ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6482024-08-14T22:28:13.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6492024-08-14T22:28:13.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6502024-08-14T22:28:13.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6512024-08-14T22:28:13.508ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6522024-08-14T22:28:13.508ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6532024-08-14T22:28:13.641ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6542024-08-14T22:28:13.641ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6552024-08-14T22:28:13.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6562024-08-14T22:28:13.711ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6572024-08-14T22:28:13.778ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6582024-08-14T22:28:13.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6592024-08-14T22:28:13.847ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6602024-08-14T22:28:13.847ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
6612024-08-14T22:28:13.847ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
6622024-08-14T22:28:13.847ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6632024-08-14T22:28:13.877ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
6642024-08-14T22:28:13.877ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
6652024-08-14T22:28:13.877ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
6662024-08-14T22:28:13.877ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
6672024-08-14T22:28:13.877ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
6682024-08-14T22:28:13.877ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 500498 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 3cead872-e205-4edb-8a14-9ffd289db9cb response_code = 500 uri = /omicron-zones
6692024-08-14T22:28:13.877ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3cead872-e205-4edb-8a14-9ffd289db9cb", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3cead872-e205-4edb-8a14-9ffd289db9cb" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 131.320169ms sled_address = [fd00:1122:3344:101::1]:12345
6702024-08-14T22:28:14.010ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6712024-08-14T22:28:14.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6722024-08-14T22:28:14.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6732024-08-14T22:28:14.029ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6742024-08-14T22:28:14.029ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6752024-08-14T22:28:14.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6762024-08-14T22:28:14.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6772024-08-14T22:28:14.065ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6782024-08-14T22:28:14.065ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6792024-08-14T22:28:14.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6802024-08-14T22:28:14.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6812024-08-14T22:28:14.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6822024-08-14T22:28:14.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6832024-08-14T22:28:14.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6842024-08-14T22:28:14.116ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6852024-08-14T22:28:14.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6862024-08-14T22:28:14.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6872024-08-14T22:28:14.151ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6882024-08-14T22:28:14.151ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
6892024-08-14T22:28:14.151ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
6902024-08-14T22:28:14.151ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6912024-08-14T22:28:14.183ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
6922024-08-14T22:28:14.183ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
6932024-08-14T22:28:14.183ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
6942024-08-14T22:28:14.183ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
6952024-08-14T22:28:14.183ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
6962024-08-14T22:28:14.183ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 172798 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = df5572f1-48e6-4dc4-9447-9100e95cea09 response_code = 500 uri = /omicron-zones
6972024-08-14T22:28:14.183ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "df5572f1-48e6-4dc4-9447-9100e95cea09", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "df5572f1-48e6-4dc4-9447-9100e95cea09" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 102.840992ms sled_address = [fd00:1122:3344:101::1]:12345
6982024-08-14T22:28:14.218ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 303.299608ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
6992024-08-14T22:28:14.287ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7002024-08-14T22:28:14.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7012024-08-14T22:28:14.288ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7022024-08-14T22:28:14.308ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7032024-08-14T22:28:14.308ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7042024-08-14T22:28:14.326ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7052024-08-14T22:28:14.326ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7062024-08-14T22:28:14.345ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7072024-08-14T22:28:14.345ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7082024-08-14T22:28:14.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7092024-08-14T22:28:14.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7102024-08-14T22:28:14.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7112024-08-14T22:28:14.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7122024-08-14T22:28:14.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7132024-08-14T22:28:14.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7142024-08-14T22:28:14.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7152024-08-14T22:28:14.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7162024-08-14T22:28:14.438ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7172024-08-14T22:28:14.438ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
7182024-08-14T22:28:14.438ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
7192024-08-14T22:28:14.438ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7202024-08-14T22:28:14.471ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
7212024-08-14T22:28:14.471ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
7222024-08-14T22:28:14.471ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
7232024-08-14T22:28:14.471ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
7242024-08-14T22:28:14.471ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
7252024-08-14T22:28:14.471ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 182920 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 94e8e277-3a77-4618-ae7e-cb4a3d41d232 response_code = 500 uri = /omicron-zones
7262024-08-14T22:28:14.471ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "94e8e277-3a77-4618-ae7e-cb4a3d41d232", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "94e8e277-3a77-4618-ae7e-cb4a3d41d232" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 130.793521ms sled_address = [fd00:1122:3344:101::1]:12345
7272024-08-14T22:28:14.604ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7282024-08-14T22:28:14.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7292024-08-14T22:28:14.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7302024-08-14T22:28:14.624ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7312024-08-14T22:28:14.624ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7322024-08-14T22:28:14.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7332024-08-14T22:28:14.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7342024-08-14T22:28:14.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7352024-08-14T22:28:14.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7362024-08-14T22:28:14.677ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7372024-08-14T22:28:14.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7382024-08-14T22:28:14.695ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7392024-08-14T22:28:14.695ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7402024-08-14T22:28:14.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7412024-08-14T22:28:14.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7422024-08-14T22:28:14.730ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7432024-08-14T22:28:14.730ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7442024-08-14T22:28:14.748ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7452024-08-14T22:28:14.748ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
7462024-08-14T22:28:14.748ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
7472024-08-14T22:28:14.748ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7482024-08-14T22:28:14.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
7492024-08-14T22:28:14.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
7502024-08-14T22:28:14.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
7512024-08-14T22:28:14.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
7522024-08-14T22:28:14.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
7532024-08-14T22:28:14.779ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 175148 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 4c0dd7f6-66bb-49d9-b65f-26ca34f9c659 response_code = 500 uri = /omicron-zones
7542024-08-14T22:28:14.780ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4c0dd7f6-66bb-49d9-b65f-26ca34f9c659", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4c0dd7f6-66bb-49d9-b65f-26ca34f9c659" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 223.13321ms sled_address = [fd00:1122:3344:101::1]:12345
7552024-08-14T22:28:15.005ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7562024-08-14T22:28:15.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7572024-08-14T22:28:15.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7582024-08-14T22:28:15.025ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7592024-08-14T22:28:15.025ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7602024-08-14T22:28:15.043ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7612024-08-14T22:28:15.043ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7622024-08-14T22:28:15.061ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7632024-08-14T22:28:15.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7642024-08-14T22:28:15.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7652024-08-14T22:28:15.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7662024-08-14T22:28:15.098ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7672024-08-14T22:28:15.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7682024-08-14T22:28:15.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7692024-08-14T22:28:15.116ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7702024-08-14T22:28:15.136ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7712024-08-14T22:28:15.136ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7722024-08-14T22:28:15.154ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7732024-08-14T22:28:15.154ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
7742024-08-14T22:28:15.154ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
7752024-08-14T22:28:15.154ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7762024-08-14T22:28:15.187ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
7772024-08-14T22:28:15.187ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
7782024-08-14T22:28:15.187ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
7792024-08-14T22:28:15.187ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
7802024-08-14T22:28:15.187ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
7812024-08-14T22:28:15.187ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 182294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = f97376ba-e1c9-4b3e-a1f9-bb5610c51da2 response_code = 500 uri = /omicron-zones
7822024-08-14T22:28:15.188ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f97376ba-e1c9-4b3e-a1f9-bb5610c51da2", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f97376ba-e1c9-4b3e-a1f9-bb5610c51da2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 196.058942ms sled_address = [fd00:1122:3344:101::1]:12345
7832024-08-14T22:28:15.386ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7842024-08-14T22:28:15.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7852024-08-14T22:28:15.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7862024-08-14T22:28:15.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7872024-08-14T22:28:15.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7882024-08-14T22:28:15.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7892024-08-14T22:28:15.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7902024-08-14T22:28:15.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7912024-08-14T22:28:15.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7922024-08-14T22:28:15.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7932024-08-14T22:28:15.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7942024-08-14T22:28:15.476ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7952024-08-14T22:28:15.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7962024-08-14T22:28:15.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7972024-08-14T22:28:15.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7982024-08-14T22:28:15.508ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7992024-08-14T22:28:15.508ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8002024-08-14T22:28:15.527ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8012024-08-14T22:28:15.527ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
8022024-08-14T22:28:15.527ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
8032024-08-14T22:28:15.527ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8042024-08-14T22:28:15.558ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
8052024-08-14T22:28:15.558ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
8062024-08-14T22:28:15.558ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
8072024-08-14T22:28:15.558ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
8082024-08-14T22:28:15.558ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
8092024-08-14T22:28:15.558ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 172023 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 441f4d41-195b-4f26-8e5e-56081eea7b2a response_code = 500 uri = /omicron-zones
8102024-08-14T22:28:15.558ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "441f4d41-195b-4f26-8e5e-56081eea7b2a", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "441f4d41-195b-4f26-8e5e-56081eea7b2a" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 241.265014ms sled_address = [fd00:1122:3344:101::1]:12345
8112024-08-14T22:28:15.801ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8122024-08-14T22:28:15.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8132024-08-14T22:28:15.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8142024-08-14T22:28:15.820ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8152024-08-14T22:28:15.820ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8162024-08-14T22:28:15.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8172024-08-14T22:28:15.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8182024-08-14T22:28:15.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8192024-08-14T22:28:15.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8202024-08-14T22:28:15.875ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8212024-08-14T22:28:15.875ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8222024-08-14T22:28:15.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8232024-08-14T22:28:15.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8242024-08-14T22:28:15.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8252024-08-14T22:28:15.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8262024-08-14T22:28:15.929ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8272024-08-14T22:28:15.929ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8282024-08-14T22:28:15.945ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8292024-08-14T22:28:15.945ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
8302024-08-14T22:28:15.945ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
8312024-08-14T22:28:15.945ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8322024-08-14T22:28:15.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
8332024-08-14T22:28:15.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
8342024-08-14T22:28:15.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
8352024-08-14T22:28:15.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
8362024-08-14T22:28:15.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
8372024-08-14T22:28:15.978ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 176476 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 9c6e24fb-89d1-43cf-ac45-7184cb3ced1d response_code = 500 uri = /omicron-zones
8382024-08-14T22:28:15.979ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9c6e24fb-89d1-43cf-ac45-7184cb3ced1d", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9c6e24fb-89d1-43cf-ac45-7184cb3ced1d" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 321.790764ms sled_address = [fd00:1122:3344:101::1]:12345
8392024-08-14T22:28:16.302ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8402024-08-14T22:28:16.303ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8412024-08-14T22:28:16.303ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8422024-08-14T22:28:16.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8432024-08-14T22:28:16.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8442024-08-14T22:28:16.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8452024-08-14T22:28:16.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8462024-08-14T22:28:16.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8472024-08-14T22:28:16.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8482024-08-14T22:28:16.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8492024-08-14T22:28:16.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8502024-08-14T22:28:16.395ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8512024-08-14T22:28:16.395ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8522024-08-14T22:28:16.413ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8532024-08-14T22:28:16.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8542024-08-14T22:28:16.431ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8552024-08-14T22:28:16.431ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8562024-08-14T22:28:16.449ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8572024-08-14T22:28:16.449ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
8582024-08-14T22:28:16.449ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
8592024-08-14T22:28:16.449ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8602024-08-14T22:28:16.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
8612024-08-14T22:28:16.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
8622024-08-14T22:28:16.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
8632024-08-14T22:28:16.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
8642024-08-14T22:28:16.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
8652024-08-14T22:28:16.481ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 178676 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 1671e03c-b55c-4849-ae28-85f254dce464 response_code = 500 uri = /omicron-zones
8662024-08-14T22:28:16.481ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1671e03c-b55c-4849-ae28-85f254dce464", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1671e03c-b55c-4849-ae28-85f254dce464" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 216.913305ms sled_address = [fd00:1122:3344:101::1]:12345
8672024-08-14T22:28:16.700ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8682024-08-14T22:28:16.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8692024-08-14T22:28:16.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8702024-08-14T22:28:16.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8712024-08-14T22:28:16.718ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8722024-08-14T22:28:16.737ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8732024-08-14T22:28:16.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8742024-08-14T22:28:16.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8752024-08-14T22:28:16.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8762024-08-14T22:28:16.772ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8772024-08-14T22:28:16.772ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8782024-08-14T22:28:16.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8792024-08-14T22:28:16.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8802024-08-14T22:28:16.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8812024-08-14T22:28:16.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8822024-08-14T22:28:16.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8832024-08-14T22:28:16.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8842024-08-14T22:28:16.844ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8852024-08-14T22:28:16.844ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
8862024-08-14T22:28:16.844ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
8872024-08-14T22:28:16.844ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8882024-08-14T22:28:16.875ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
8892024-08-14T22:28:16.875ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
8902024-08-14T22:28:16.875ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
8912024-08-14T22:28:16.875ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
8922024-08-14T22:28:16.875ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
8932024-08-14T22:28:16.875ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 174163 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = a1722792-f017-4e1a-a846-acea29702e13 response_code = 500 uri = /omicron-zones
8942024-08-14T22:28:16.875ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a1722792-f017-4e1a-a846-acea29702e13", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a1722792-f017-4e1a-a846-acea29702e13" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 489.814991ms sled_address = [fd00:1122:3344:101::1]:12345
8952024-08-14T22:28:17.366ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8962024-08-14T22:28:17.368ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8972024-08-14T22:28:17.368ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8982024-08-14T22:28:17.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8992024-08-14T22:28:17.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9002024-08-14T22:28:17.406ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9012024-08-14T22:28:17.406ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9022024-08-14T22:28:17.423ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9032024-08-14T22:28:17.423ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9042024-08-14T22:28:17.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9052024-08-14T22:28:17.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9062024-08-14T22:28:17.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9072024-08-14T22:28:17.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9082024-08-14T22:28:17.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9092024-08-14T22:28:17.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9102024-08-14T22:28:17.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9112024-08-14T22:28:17.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9122024-08-14T22:28:17.512ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9132024-08-14T22:28:17.512ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
9142024-08-14T22:28:17.512ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
9152024-08-14T22:28:17.512ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9162024-08-14T22:28:17.543ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
9172024-08-14T22:28:17.543ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
9182024-08-14T22:28:17.543ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
9192024-08-14T22:28:17.543ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
9202024-08-14T22:28:17.543ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
9212024-08-14T22:28:17.543ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 175790 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 1f9eea40-d35c-424d-9b53-345c8f12bb15 response_code = 500 uri = /omicron-zones
9222024-08-14T22:28:17.543ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1f9eea40-d35c-424d-9b53-345c8f12bb15", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1f9eea40-d35c-424d-9b53-345c8f12bb15" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 275.080841ms sled_address = [fd00:1122:3344:101::1]:12345
9232024-08-14T22:28:17.820ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9242024-08-14T22:28:17.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9252024-08-14T22:28:17.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9262024-08-14T22:28:17.839ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9272024-08-14T22:28:17.839ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9282024-08-14T22:28:17.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9292024-08-14T22:28:17.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9302024-08-14T22:28:17.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9312024-08-14T22:28:17.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9322024-08-14T22:28:17.889ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9332024-08-14T22:28:17.889ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9342024-08-14T22:28:17.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9352024-08-14T22:28:17.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9362024-08-14T22:28:17.925ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9372024-08-14T22:28:17.925ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9382024-08-14T22:28:17.943ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9392024-08-14T22:28:17.943ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9402024-08-14T22:28:17.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9412024-08-14T22:28:17.959ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
9422024-08-14T22:28:17.959ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
9432024-08-14T22:28:17.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9442024-08-14T22:28:17.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
9452024-08-14T22:28:17.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
9462024-08-14T22:28:17.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
9472024-08-14T22:28:17.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
9482024-08-14T22:28:17.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
9492024-08-14T22:28:17.989ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 168332 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 35428f21-3028-4ab1-8a07-a094c671ee94 response_code = 500 uri = /omicron-zones
9502024-08-14T22:28:17.989ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "35428f21-3028-4ab1-8a07-a094c671ee94", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "35428f21-3028-4ab1-8a07-a094c671ee94" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 745.408633ms sled_address = [fd00:1122:3344:101::1]:12345
9512024-08-14T22:28:18.737ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9522024-08-14T22:28:18.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9532024-08-14T22:28:18.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9542024-08-14T22:28:18.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9552024-08-14T22:28:18.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9562024-08-14T22:28:18.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9572024-08-14T22:28:18.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9582024-08-14T22:28:18.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9592024-08-14T22:28:18.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9602024-08-14T22:28:18.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9612024-08-14T22:28:18.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9622024-08-14T22:28:18.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9632024-08-14T22:28:18.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9642024-08-14T22:28:18.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9652024-08-14T22:28:18.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9662024-08-14T22:28:18.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9672024-08-14T22:28:18.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9682024-08-14T22:28:18.887ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9692024-08-14T22:28:18.887ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
9702024-08-14T22:28:18.887ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
9712024-08-14T22:28:18.887ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9722024-08-14T22:28:18.920ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
9732024-08-14T22:28:18.920ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
9742024-08-14T22:28:18.920ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
9752024-08-14T22:28:18.920ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
9762024-08-14T22:28:18.920ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
9772024-08-14T22:28:18.920ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 181826 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 18aee285-a932-475e-b0bf-a9afb246b17a response_code = 500 uri = /omicron-zones
9782024-08-14T22:28:18.920ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "18aee285-a932-475e-b0bf-a9afb246b17a", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "18aee285-a932-475e-b0bf-a9afb246b17a" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.094041744s sled_address = [fd00:1122:3344:101::1]:12345
9792024-08-14T22:28:20.018ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9802024-08-14T22:28:20.019ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9812024-08-14T22:28:20.019ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9822024-08-14T22:28:20.039ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9832024-08-14T22:28:20.039ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9842024-08-14T22:28:20.058ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9852024-08-14T22:28:20.058ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9862024-08-14T22:28:20.076ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9872024-08-14T22:28:20.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9882024-08-14T22:28:20.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9892024-08-14T22:28:20.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9902024-08-14T22:28:20.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9912024-08-14T22:28:20.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9922024-08-14T22:28:20.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9932024-08-14T22:28:20.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9942024-08-14T22:28:20.152ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9952024-08-14T22:28:20.152ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9962024-08-14T22:28:20.171ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9972024-08-14T22:28:20.171ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
9982024-08-14T22:28:20.171ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
9992024-08-14T22:28:20.171ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10002024-08-14T22:28:20.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
10012024-08-14T22:28:20.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
10022024-08-14T22:28:20.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
10032024-08-14T22:28:20.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
10042024-08-14T22:28:20.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
10052024-08-14T22:28:20.205ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 186221 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 5c75158a-f407-4d0d-9a15-353095e09ab2 response_code = 500 uri = /omicron-zones
10062024-08-14T22:28:20.205ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5c75158a-f407-4d0d-9a15-353095e09ab2", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5c75158a-f407-4d0d-9a15-353095e09ab2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 619.607096ms sled_address = [fd00:1122:3344:101::1]:12345
10072024-08-14T22:28:20.827ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10082024-08-14T22:28:20.827ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10092024-08-14T22:28:20.827ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10102024-08-14T22:28:20.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10112024-08-14T22:28:20.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10122024-08-14T22:28:20.865ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10132024-08-14T22:28:20.865ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10142024-08-14T22:28:20.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10152024-08-14T22:28:20.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10162024-08-14T22:28:20.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10172024-08-14T22:28:20.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10182024-08-14T22:28:20.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10192024-08-14T22:28:20.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10202024-08-14T22:28:20.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10212024-08-14T22:28:20.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10222024-08-14T22:28:20.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10232024-08-14T22:28:20.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10242024-08-14T22:28:20.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10252024-08-14T22:28:20.975ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
10262024-08-14T22:28:20.975ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
10272024-08-14T22:28:20.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10282024-08-14T22:28:21.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
10292024-08-14T22:28:21.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
10302024-08-14T22:28:21.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
10312024-08-14T22:28:21.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
10322024-08-14T22:28:21.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
10332024-08-14T22:28:21.007ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 179465 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = aac5c2c0-8213-425a-a953-52e43b4bbef4 response_code = 500 uri = /omicron-zones
10342024-08-14T22:28:21.007ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "aac5c2c0-8213-425a-a953-52e43b4bbef4", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "aac5c2c0-8213-425a-a953-52e43b4bbef4" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.57465962s sled_address = [fd00:1122:3344:101::1]:12345
10352024-08-14T22:28:22.587ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10362024-08-14T22:28:22.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10372024-08-14T22:28:22.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10382024-08-14T22:28:22.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10392024-08-14T22:28:22.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10402024-08-14T22:28:22.626ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10412024-08-14T22:28:22.626ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10422024-08-14T22:28:22.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10432024-08-14T22:28:22.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10442024-08-14T22:28:22.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10452024-08-14T22:28:22.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10462024-08-14T22:28:22.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10472024-08-14T22:28:22.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10482024-08-14T22:28:22.702ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10492024-08-14T22:28:22.702ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10502024-08-14T22:28:22.721ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10512024-08-14T22:28:22.721ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10522024-08-14T22:28:22.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10532024-08-14T22:28:22.741ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
10542024-08-14T22:28:22.741ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
10552024-08-14T22:28:22.741ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10562024-08-14T22:28:22.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
10572024-08-14T22:28:22.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
10582024-08-14T22:28:22.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
10592024-08-14T22:28:22.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
10602024-08-14T22:28:22.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
10612024-08-14T22:28:22.773ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 185319 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 67d3bf99-1cb2-4a74-a090-a8b63566e84d response_code = 500 uri = /omicron-zones
10622024-08-14T22:28:22.773ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "67d3bf99-1cb2-4a74-a090-a8b63566e84d", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "67d3bf99-1cb2-4a74-a090-a8b63566e84d" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.876592443s sled_address = [fd00:1122:3344:101::1]:12345
10632024-08-14T22:28:23.322ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41265
10642024-08-14T22:28:23.322ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41265 req_id = b9a0c948-42ca-41d1-a4c7-1745374438ac response_code = 200 uri = /baseboard
10652024-08-14T22:28:24.655ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10662024-08-14T22:28:24.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10672024-08-14T22:28:24.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10682024-08-14T22:28:24.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10692024-08-14T22:28:24.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10702024-08-14T22:28:24.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10712024-08-14T22:28:24.693ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10722024-08-14T22:28:24.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10732024-08-14T22:28:24.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10742024-08-14T22:28:24.730ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10752024-08-14T22:28:24.730ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10762024-08-14T22:28:24.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10772024-08-14T22:28:24.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10782024-08-14T22:28:24.767ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10792024-08-14T22:28:24.767ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10802024-08-14T22:28:24.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10812024-08-14T22:28:24.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10822024-08-14T22:28:24.801ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10832024-08-14T22:28:24.801ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
10842024-08-14T22:28:24.801ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
10852024-08-14T22:28:24.801ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10862024-08-14T22:28:24.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
10872024-08-14T22:28:24.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
10882024-08-14T22:28:24.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
10892024-08-14T22:28:24.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
10902024-08-14T22:28:24.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
10912024-08-14T22:28:24.833ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 176993 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = c5204a71-1b79-416f-a63e-333a23477677 response_code = 500 uri = /omicron-zones
10922024-08-14T22:28:24.833ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c5204a71-1b79-416f-a63e-333a23477677", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c5204a71-1b79-416f-a63e-333a23477677" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.585687656s sled_address = [fd00:1122:3344:101::1]:12345
10932024-08-14T22:28:26.424ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10942024-08-14T22:28:26.425ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10952024-08-14T22:28:26.425ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10962024-08-14T22:28:26.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10972024-08-14T22:28:26.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10982024-08-14T22:28:26.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10992024-08-14T22:28:26.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11002024-08-14T22:28:26.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11012024-08-14T22:28:26.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11022024-08-14T22:28:26.500ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11032024-08-14T22:28:26.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11042024-08-14T22:28:26.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11052024-08-14T22:28:26.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11062024-08-14T22:28:26.540ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11072024-08-14T22:28:26.540ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11082024-08-14T22:28:26.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11092024-08-14T22:28:26.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11102024-08-14T22:28:26.578ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11112024-08-14T22:28:26.578ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
11122024-08-14T22:28:26.578ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
11132024-08-14T22:28:26.578ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11142024-08-14T22:28:26.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
11152024-08-14T22:28:26.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
11162024-08-14T22:28:26.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
11172024-08-14T22:28:26.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
11182024-08-14T22:28:26.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
11192024-08-14T22:28:26.611ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 186258 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = b420393a-2748-4a2e-bdfa-57d75f805e42 response_code = 500 uri = /omicron-zones
11202024-08-14T22:28:26.611ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b420393a-2748-4a2e-bdfa-57d75f805e42", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:26 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b420393a-2748-4a2e-bdfa-57d75f805e42" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.499913282s sled_address = [fd00:1122:3344:101::1]:12345
11212024-08-14T22:28:28.116ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11222024-08-14T22:28:28.117ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11232024-08-14T22:28:28.117ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11242024-08-14T22:28:28.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11252024-08-14T22:28:28.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11262024-08-14T22:28:28.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11272024-08-14T22:28:28.155ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11282024-08-14T22:28:28.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11292024-08-14T22:28:28.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11302024-08-14T22:28:28.190ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11312024-08-14T22:28:28.190ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11322024-08-14T22:28:28.208ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11332024-08-14T22:28:28.208ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11342024-08-14T22:28:28.226ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11352024-08-14T22:28:28.226ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11362024-08-14T22:28:28.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11372024-08-14T22:28:28.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11382024-08-14T22:28:28.263ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11392024-08-14T22:28:28.263ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
11402024-08-14T22:28:28.263ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
11412024-08-14T22:28:28.263ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11422024-08-14T22:28:28.296ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
11432024-08-14T22:28:28.296ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
11442024-08-14T22:28:28.296ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
11452024-08-14T22:28:28.296ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
11462024-08-14T22:28:28.296ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
11472024-08-14T22:28:28.296ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 179169 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 38e8040d-c9d1-4106-8d46-282307feba55 response_code = 500 uri = /omicron-zones
11482024-08-14T22:28:28.296ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "38e8040d-c9d1-4106-8d46-282307feba55", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:28 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "38e8040d-c9d1-4106-8d46-282307feba55" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.904570598s sled_address = [fd00:1122:3344:101::1]:12345
11492024-08-14T22:28:29.599ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 348.796973ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
11502024-08-14T22:28:30.207ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11512024-08-14T22:28:30.208ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11522024-08-14T22:28:30.208ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11532024-08-14T22:28:30.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11542024-08-14T22:28:30.228ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11552024-08-14T22:28:30.246ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11562024-08-14T22:28:30.246ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11572024-08-14T22:28:30.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11582024-08-14T22:28:30.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11592024-08-14T22:28:30.284ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11602024-08-14T22:28:30.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11612024-08-14T22:28:30.304ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11622024-08-14T22:28:30.304ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11632024-08-14T22:28:30.323ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11642024-08-14T22:28:30.323ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11652024-08-14T22:28:30.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11662024-08-14T22:28:30.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11672024-08-14T22:28:30.362ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11682024-08-14T22:28:30.362ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
11692024-08-14T22:28:30.362ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
11702024-08-14T22:28:30.362ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11712024-08-14T22:28:30.396ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
11722024-08-14T22:28:30.396ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
11732024-08-14T22:28:30.396ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
11742024-08-14T22:28:30.396ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
11752024-08-14T22:28:30.396ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
11762024-08-14T22:28:30.396ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 188288 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 5e780bc7-87b5-42d7-b8d1-ef0cf84723b8 response_code = 500 uri = /omicron-zones
11772024-08-14T22:28:30.396ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5e780bc7-87b5-42d7-b8d1-ef0cf84723b8", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5e780bc7-87b5-42d7-b8d1-ef0cf84723b8" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.384629258s sled_address = [fd00:1122:3344:101::1]:12345
11782024-08-14T22:28:33.792ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11792024-08-14T22:28:33.792ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11802024-08-14T22:28:33.792ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11812024-08-14T22:28:33.814ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11822024-08-14T22:28:33.814ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11832024-08-14T22:28:33.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11842024-08-14T22:28:33.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11852024-08-14T22:28:33.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11862024-08-14T22:28:33.852ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11872024-08-14T22:28:33.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11882024-08-14T22:28:33.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11892024-08-14T22:28:33.891ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11902024-08-14T22:28:33.891ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11912024-08-14T22:28:33.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11922024-08-14T22:28:33.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11932024-08-14T22:28:33.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11942024-08-14T22:28:33.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11952024-08-14T22:28:33.946ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11962024-08-14T22:28:33.946ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
11972024-08-14T22:28:33.946ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
11982024-08-14T22:28:33.946ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11992024-08-14T22:28:33.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
12002024-08-14T22:28:33.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
12012024-08-14T22:28:33.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
12022024-08-14T22:28:33.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
12032024-08-14T22:28:33.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
12042024-08-14T22:28:33.978ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 185904 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 0488b99f-e4b1-4315-b5e5-e59c31d78cde response_code = 500 uri = /omicron-zones
12052024-08-14T22:28:33.979ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0488b99f-e4b1-4315-b5e5-e59c31d78cde", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0488b99f-e4b1-4315-b5e5-e59c31d78cde" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.45652494s sled_address = [fd00:1122:3344:101::1]:12345
12062024-08-14T22:28:36.443ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12072024-08-14T22:28:36.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12082024-08-14T22:28:36.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12092024-08-14T22:28:36.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12102024-08-14T22:28:36.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12112024-08-14T22:28:36.484ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12122024-08-14T22:28:36.484ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12132024-08-14T22:28:36.502ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12142024-08-14T22:28:36.502ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12152024-08-14T22:28:36.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12162024-08-14T22:28:36.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12172024-08-14T22:28:36.541ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12182024-08-14T22:28:36.541ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12192024-08-14T22:28:36.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12202024-08-14T22:28:36.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12212024-08-14T22:28:36.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12222024-08-14T22:28:36.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12232024-08-14T22:28:36.597ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12242024-08-14T22:28:36.597ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
12252024-08-14T22:28:36.597ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
12262024-08-14T22:28:36.597ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12272024-08-14T22:28:36.630ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
12282024-08-14T22:28:36.630ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
12292024-08-14T22:28:36.630ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
12302024-08-14T22:28:36.630ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
12312024-08-14T22:28:36.630ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
12322024-08-14T22:28:36.631ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 187068 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = e85aa7bf-3c0d-4df8-b008-69e3d12614b5 response_code = 500 uri = /omicron-zones
12332024-08-14T22:28:36.631ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e85aa7bf-3c0d-4df8-b008-69e3d12614b5", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e85aa7bf-3c0d-4df8-b008-69e3d12614b5" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.74850362s sled_address = [fd00:1122:3344:101::1]:12345
12342024-08-14T22:28:41.393ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12352024-08-14T22:28:41.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12362024-08-14T22:28:41.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12372024-08-14T22:28:41.414ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12382024-08-14T22:28:41.414ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12392024-08-14T22:28:41.431ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12402024-08-14T22:28:41.431ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12412024-08-14T22:28:41.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12422024-08-14T22:28:41.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12432024-08-14T22:28:41.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12442024-08-14T22:28:41.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12452024-08-14T22:28:41.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12462024-08-14T22:28:41.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12472024-08-14T22:28:41.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12482024-08-14T22:28:41.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12492024-08-14T22:28:41.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12502024-08-14T22:28:41.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12512024-08-14T22:28:41.536ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12522024-08-14T22:28:41.536ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
12532024-08-14T22:28:41.537ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
12542024-08-14T22:28:41.537ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12552024-08-14T22:28:41.568ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
12562024-08-14T22:28:41.568ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
12572024-08-14T22:28:41.568ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
12582024-08-14T22:28:41.568ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
12592024-08-14T22:28:41.568ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
12602024-08-14T22:28:41.568ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 174905 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 62ae35cf-53b8-4f53-8118-777afc0bab45 response_code = 500 uri = /omicron-zones
12612024-08-14T22:28:41.569ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "62ae35cf-53b8-4f53-8118-777afc0bab45", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "62ae35cf-53b8-4f53-8118-777afc0bab45" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.999986808s sled_address = [fd00:1122:3344:101::1]:12345
12622024-08-14T22:28:45.025ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 767.756767ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
12632024-08-14T22:28:45.581ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12642024-08-14T22:28:45.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12652024-08-14T22:28:45.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12662024-08-14T22:28:45.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12672024-08-14T22:28:45.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12682024-08-14T22:28:45.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12692024-08-14T22:28:45.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12702024-08-14T22:28:45.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12712024-08-14T22:28:45.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12722024-08-14T22:28:45.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12732024-08-14T22:28:45.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12742024-08-14T22:28:45.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12752024-08-14T22:28:45.676ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12762024-08-14T22:28:45.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12772024-08-14T22:28:45.693ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12782024-08-14T22:28:45.709ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12792024-08-14T22:28:45.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12802024-08-14T22:28:45.727ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12812024-08-14T22:28:45.727ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
12822024-08-14T22:28:45.728ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
12832024-08-14T22:28:45.728ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12842024-08-14T22:28:45.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
12852024-08-14T22:28:45.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
12862024-08-14T22:28:45.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
12872024-08-14T22:28:45.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
12882024-08-14T22:28:45.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
12892024-08-14T22:28:45.762ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 180638 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 1dc6844c-4d41-4265-a04b-7fc9010fbcac response_code = 500 uri = /omicron-zones
12902024-08-14T22:28:45.762ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1dc6844c-4d41-4265-a04b-7fc9010fbcac", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1dc6844c-4d41-4265-a04b-7fc9010fbcac" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.299090931s sled_address = [fd00:1122:3344:101::1]:12345
12912024-08-14T22:28:49.071ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12922024-08-14T22:28:49.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12932024-08-14T22:28:49.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12942024-08-14T22:28:49.093ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12952024-08-14T22:28:49.093ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12962024-08-14T22:28:49.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12972024-08-14T22:28:49.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12982024-08-14T22:28:49.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12992024-08-14T22:28:49.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13002024-08-14T22:28:49.149ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13012024-08-14T22:28:49.149ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13022024-08-14T22:28:49.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13032024-08-14T22:28:49.167ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13042024-08-14T22:28:49.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13052024-08-14T22:28:49.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13062024-08-14T22:28:49.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13072024-08-14T22:28:49.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13082024-08-14T22:28:49.221ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13092024-08-14T22:28:49.221ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
13102024-08-14T22:28:49.221ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
13112024-08-14T22:28:49.221ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13122024-08-14T22:28:49.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
13132024-08-14T22:28:49.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
13142024-08-14T22:28:49.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
13152024-08-14T22:28:49.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
13162024-08-14T22:28:49.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
13172024-08-14T22:28:49.253ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 181149 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = a34da9f2-d109-4b56-a51e-ef30ab52d368 response_code = 500 uri = /omicron-zones
13182024-08-14T22:28:49.253ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a34da9f2-d109-4b56-a51e-ef30ab52d368", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a34da9f2-d109-4b56-a51e-ef30ab52d368" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 5.287957528s sled_address = [fd00:1122:3344:101::1]:12345
13192024-08-14T22:28:53.437ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56184
13202024-08-14T22:28:53.437ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56184 req_id = 59b7c839-0ddb-4ff0-a727-f0de6b4b8a9f response_code = 200 uri = /baseboard
13212024-08-14T22:28:54.557ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13222024-08-14T22:28:54.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13232024-08-14T22:28:54.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13242024-08-14T22:28:54.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13252024-08-14T22:28:54.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13262024-08-14T22:28:54.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13272024-08-14T22:28:54.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13282024-08-14T22:28:54.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13292024-08-14T22:28:54.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13302024-08-14T22:28:54.638ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13312024-08-14T22:28:54.638ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13322024-08-14T22:28:54.657ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13332024-08-14T22:28:54.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13342024-08-14T22:28:54.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13352024-08-14T22:28:54.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13362024-08-14T22:28:54.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13372024-08-14T22:28:54.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13382024-08-14T22:28:54.712ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13392024-08-14T22:28:54.712ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
13402024-08-14T22:28:54.712ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
13412024-08-14T22:28:54.712ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13422024-08-14T22:28:54.745ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
13432024-08-14T22:28:54.745ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
13442024-08-14T22:28:54.745ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
13452024-08-14T22:28:54.745ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
13462024-08-14T22:28:54.745ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
13472024-08-14T22:28:54.745ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 188043 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = bb0e576b-137f-4e1a-afd0-a44829f38db9 response_code = 500 uri = /omicron-zones
13482024-08-14T22:28:54.746ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bb0e576b-137f-4e1a-afd0-a44829f38db9", "content-length": "124", "date": "Wed, 14 Aug 2024 22:28:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bb0e576b-137f-4e1a-afd0-a44829f38db9" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 5.38076024s sled_address = [fd00:1122:3344:101::1]:12345
13492024-08-14T22:29:00.142ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13502024-08-14T22:29:00.142ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13512024-08-14T22:29:00.142ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13522024-08-14T22:29:00.163ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13532024-08-14T22:29:00.163ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13542024-08-14T22:29:00.183ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13552024-08-14T22:29:00.183ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13562024-08-14T22:29:00.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13572024-08-14T22:29:00.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13582024-08-14T22:29:00.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13592024-08-14T22:29:00.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13602024-08-14T22:29:00.240ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13612024-08-14T22:29:00.240ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13622024-08-14T22:29:00.258ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13632024-08-14T22:29:00.258ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13642024-08-14T22:29:00.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13652024-08-14T22:29:00.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13662024-08-14T22:29:00.295ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13672024-08-14T22:29:00.295ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
13682024-08-14T22:29:00.295ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
13692024-08-14T22:29:00.295ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13702024-08-14T22:29:00.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
13712024-08-14T22:29:00.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
13722024-08-14T22:29:00.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
13732024-08-14T22:29:00.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
13742024-08-14T22:29:00.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
13752024-08-14T22:29:00.328ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 185450 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 73998b3c-f55f-4d10-b517-0b39c5d2ad7c response_code = 500 uri = /omicron-zones
13762024-08-14T22:29:00.328ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "73998b3c-f55f-4d10-b517-0b39c5d2ad7c", "content-length": "124", "date": "Wed, 14 Aug 2024 22:29:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "73998b3c-f55f-4d10-b517-0b39c5d2ad7c" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 14.017506449s sled_address = [fd00:1122:3344:101::1]:12345
13772024-08-14T22:29:00.872ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.385512302s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
13782024-08-14T22:29:14.385ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13792024-08-14T22:29:14.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13802024-08-14T22:29:14.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13812024-08-14T22:29:14.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13822024-08-14T22:29:14.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13832024-08-14T22:29:14.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13842024-08-14T22:29:14.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13852024-08-14T22:29:14.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13862024-08-14T22:29:14.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13872024-08-14T22:29:14.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13882024-08-14T22:29:14.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13892024-08-14T22:29:14.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13902024-08-14T22:29:14.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13912024-08-14T22:29:14.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13922024-08-14T22:29:14.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13932024-08-14T22:29:14.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13942024-08-14T22:29:14.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13952024-08-14T22:29:14.542ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13962024-08-14T22:29:14.542ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
13972024-08-14T22:29:14.542ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
13982024-08-14T22:29:14.542ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13992024-08-14T22:29:14.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
14002024-08-14T22:29:14.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
14012024-08-14T22:29:14.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
14022024-08-14T22:29:14.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
14032024-08-14T22:29:14.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
14042024-08-14T22:29:14.575ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 188866 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = ba6f274f-a8c3-4358-b857-24c8546b283f response_code = 500 uri = /omicron-zones
14052024-08-14T22:29:14.575ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ba6f274f-a8c3-4358-b857-24c8546b283f", "content-length": "124", "date": "Wed, 14 Aug 2024 22:29:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ba6f274f-a8c3-4358-b857-24c8546b283f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 12.911074627s sled_address = [fd00:1122:3344:101::1]:12345
14062024-08-14T22:29:18.341ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.914959829s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
14072024-08-14T22:29:23.557ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57824
14082024-08-14T22:29:23.557ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 31 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57824 req_id = a876b8f8-e3bb-49e0-87d7-ea0c755c6020 response_code = 200 uri = /baseboard
14092024-08-14T22:29:27.523ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14102024-08-14T22:29:27.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14112024-08-14T22:29:27.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14122024-08-14T22:29:27.545ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14132024-08-14T22:29:27.545ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14142024-08-14T22:29:27.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14152024-08-14T22:29:27.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14162024-08-14T22:29:27.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14172024-08-14T22:29:27.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14182024-08-14T22:29:27.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14192024-08-14T22:29:27.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14202024-08-14T22:29:27.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14212024-08-14T22:29:27.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14222024-08-14T22:29:27.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14232024-08-14T22:29:27.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14242024-08-14T22:29:27.659ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14252024-08-14T22:29:27.659ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14262024-08-14T22:29:27.677ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14272024-08-14T22:29:27.677ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
14282024-08-14T22:29:27.678ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
14292024-08-14T22:29:27.678ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14302024-08-14T22:29:27.711ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
14312024-08-14T22:29:27.711ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
14322024-08-14T22:29:27.711ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
14332024-08-14T22:29:27.711ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
14342024-08-14T22:29:27.711ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
14352024-08-14T22:29:27.711ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 188225 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 28436048-c807-4da0-adca-8f99c8811bee response_code = 500 uri = /omicron-zones
14362024-08-14T22:29:27.712ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "28436048-c807-4da0-adca-8f99c8811bee", "content-length": "124", "date": "Wed, 14 Aug 2024 22:29:27 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "28436048-c807-4da0-adca-8f99c8811bee" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 15.547192305s sled_address = [fd00:1122:3344:101::1]:12345
14372024-08-14T22:29:37.343ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.421591553s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
14382024-08-14T22:29:43.302ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14392024-08-14T22:29:43.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14402024-08-14T22:29:43.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14412024-08-14T22:29:43.324ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14422024-08-14T22:29:43.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14432024-08-14T22:29:43.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14442024-08-14T22:29:43.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14452024-08-14T22:29:43.361ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14462024-08-14T22:29:43.361ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14472024-08-14T22:29:43.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14482024-08-14T22:29:43.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14492024-08-14T22:29:43.397ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14502024-08-14T22:29:43.397ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14512024-08-14T22:29:43.416ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14522024-08-14T22:29:43.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14532024-08-14T22:29:43.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14542024-08-14T22:29:43.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14552024-08-14T22:29:43.453ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14562024-08-14T22:29:43.453ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
14572024-08-14T22:29:43.453ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
14582024-08-14T22:29:43.453ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14592024-08-14T22:29:43.488ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
14602024-08-14T22:29:43.488ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
14612024-08-14T22:29:43.488ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
14622024-08-14T22:29:43.488ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
14632024-08-14T22:29:43.488ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3166 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
14642024-08-14T22:29:43.488ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050", TimeNotSynchronized), ("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b", TimeNotSynchronized), ("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a", TimeNotSynchronized), ("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e", TimeNotSynchronized), ("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:902 latency_us = 185337 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 17561f06-241a-4e1a-95c9-a7b7f508a724 response_code = 500 uri = /omicron-zones
14652024-08-14T22:29:43.488ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "17561f06-241a-4e1a-95c9-a7b7f508a724", "content-length": "124", "date": "Wed, 14 Aug 2024 22:29:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "17561f06-241a-4e1a-95c9-a7b7f508a724" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 22.198528028s sled_address = [fd00:1122:3344:101::1]:12345
14662024-08-14T22:29:53.670ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40551
14672024-08-14T22:29:53.670ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40551 req_id = 75607085-fc53-4413-b3c0-dc6616b6a1d8 response_code = 200 uri = /baseboard
14682024-08-14T22:30:01.866ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.846901206s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
14692024-08-14T22:30:05.748ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14702024-08-14T22:30:05.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14712024-08-14T22:30:05.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14722024-08-14T22:30:05.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14732024-08-14T22:30:05.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14742024-08-14T22:30:05.787ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14752024-08-14T22:30:05.787ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14762024-08-14T22:30:05.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14772024-08-14T22:30:05.803ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14782024-08-14T22:30:05.824ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14792024-08-14T22:30:05.824ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14802024-08-14T22:30:05.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14812024-08-14T22:30:05.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14822024-08-14T22:30:05.859ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14832024-08-14T22:30:05.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14842024-08-14T22:30:05.877ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14852024-08-14T22:30:05.877ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14862024-08-14T22:30:05.894ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14872024-08-14T22:30:05.894ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
14882024-08-14T22:30:05.895ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
14892024-08-14T22:30:05.895ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14902024-08-14T22:30:05.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14912024-08-14T22:30:05.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14922024-08-14T22:30:05.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14932024-08-14T22:30:05.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14942024-08-14T22:30:05.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14952024-08-14T22:30:06.061ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e file = illumos-utils/src/zone.rs:316
14962024-08-14T22:30:06.064ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050 file = illumos-utils/src/zone.rs:316
14972024-08-14T22:30:06.069ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b file = illumos-utils/src/zone.rs:316
14982024-08-14T22:30:06.073ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a file = illumos-utils/src/zone.rs:316
14992024-08-14T22:30:06.078ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876 file = illumos-utils/src/zone.rs:316
15002024-08-14T22:30:06.082ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e file = illumos-utils/src/zone.rs:354
15012024-08-14T22:30:06.088ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050 file = illumos-utils/src/zone.rs:354
15022024-08-14T22:30:06.093ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b file = illumos-utils/src/zone.rs:354
15032024-08-14T22:30:06.098ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a file = illumos-utils/src/zone.rs:354
15042024-08-14T22:30:06.103ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876 file = illumos-utils/src/zone.rs:354
15052024-08-14T22:30:09.014ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='c7a204e7-7a9e-433d-a12b-a29fd64ac00e'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15062024-08-14T22:30:09.014ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15072024-08-14T22:30:09.021ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='734a7da0-4664-48e8-b4a5-4a758b3f3876'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15082024-08-14T22:30:09.021ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15092024-08-14T22:30:09.086ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='4f9822a7-fb64-419c-8824-2c8538ed9a6a'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15102024-08-14T22:30:09.086ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15112024-08-14T22:30:09.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 44.5965ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15122024-08-14T22:30:09.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 40.891477ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15132024-08-14T22:30:09.178ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='096736ee-173c-4ab4-9218-85c2f4818050'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15142024-08-14T22:30:09.178ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15152024-08-14T22:30:09.187ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15162024-08-14T22:30:09.187ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15172024-08-14T22:30:09.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 55.206141ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15182024-08-14T22:30:09.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 73.778084ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15192024-08-14T22:30:09.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 47.905851ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15202024-08-14T22:30:09.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 255.37418ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15212024-08-14T22:30:09.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 116.634433ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15222024-08-14T22:30:09.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 216.774661ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15232024-08-14T22:30:10.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 509.463492ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15242024-08-14T22:30:10.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 175.643706ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15252024-08-14T22:30:10.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 369.167277ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15262024-08-14T22:30:10.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 37.474972ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15272024-08-14T22:30:10.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 458.901021ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15282024-08-14T22:30:10.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 52.419737ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15292024-08-14T22:30:10.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 60.007902ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15302024-08-14T22:30:10.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 577.936331ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15312024-08-14T22:30:10.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.002017438s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15322024-08-14T22:30:10.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 277.025123ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15332024-08-14T22:30:10.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 64.36528ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15342024-08-14T22:30:10.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 656.951765ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15352024-08-14T22:30:10.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 291.667326ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15362024-08-14T22:30:11.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 593.015885ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15372024-08-14T22:30:11.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 574.160683ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15382024-08-14T22:30:11.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 1.090388518s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15392024-08-14T22:30:11.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 670.71036ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15402024-08-14T22:30:11.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 940.384373ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15412024-08-14T22:30:11.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.193841532s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15422024-08-14T22:30:12.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 746.104403ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15432024-08-14T22:30:12.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 1.145091582s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15442024-08-14T22:30:12.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 602.842073ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15452024-08-14T22:30:12.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 1.113859328s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15462024-08-14T22:30:12.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 882.997622ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15472024-08-14T22:30:13.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 765.802827ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15482024-08-14T22:30:13.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.119345568s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15492024-08-14T22:30:13.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 660.449944ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15502024-08-14T22:30:13.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 880.433404ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15512024-08-14T22:30:13.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 829.361045ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15522024-08-14T22:30:13.909ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 579.619512ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15532024-08-14T22:30:14.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 599.873213ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15542024-08-14T22:30:14.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 517.898946ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15552024-08-14T22:30:14.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 1.296823948s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15562024-08-14T22:30:14.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 1.333153093s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15572024-08-14T22:30:14.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 1.17763796s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15582024-08-14T22:30:14.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 662.527856ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15592024-08-14T22:30:14.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.078195102s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15602024-08-14T22:30:15.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 1.269682069s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15612024-08-14T22:30:15.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 1.485963644s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15622024-08-14T22:30:15.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.176195575s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15632024-08-14T22:30:15.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 529.840632ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15642024-08-14T22:30:15.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 1.101850684s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15652024-08-14T22:30:16.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 1.451808755s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15662024-08-14T22:30:16.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 767.931788ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15672024-08-14T22:30:17.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.178846249s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15682024-08-14T22:30:17.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 661.488004ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15692024-08-14T22:30:17.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050") failed: Property not found. retry in 1.179786192s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15702024-08-14T22:30:17.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a") failed: Property not found. retry in 811.47317ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15712024-08-14T22:30:17.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b") failed: Property not found. retry in 840.238256ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15722024-08-14T22:30:17.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876") failed: Property not found. retry in 834.091609ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15732024-08-14T22:30:18.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 789.13285ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15742024-08-14T22:30:19.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.092926623s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15752024-08-14T22:30:20.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.047009075s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15762024-08-14T22:30:21.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.205730363s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15772024-08-14T22:30:22.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.431099574s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15782024-08-14T22:30:23.794ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33141
15792024-08-14T22:30:23.795ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1151 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33141 req_id = 04d19926-d1cd-49f6-a4b6-82eedc5ccca2 response_code = 200 uri = /baseboard
15802024-08-14T22:30:24.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 889.975778ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15812024-08-14T22:30:25.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 981.404858ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15822024-08-14T22:30:26.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 1.369035953s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15832024-08-14T22:30:26.841ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 29.718243066s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
15842024-08-14T22:30:27.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e") failed: Property not found. retry in 978.387007ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15852024-08-14T22:30:28.580ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_cockroachdb_c7a204e7-7a9e-433d-a12b-a29fd64ac00e
15862024-08-14T22:30:28.581ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_cockroachdb_096736ee-173c-4ab4-9218-85c2f4818050
15872024-08-14T22:30:28.581ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_cockroachdb_e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b
15882024-08-14T22:30:28.581ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_cockroachdb_4f9822a7-fb64-419c-8824-2c8538ed9a6a
15892024-08-14T22:30:28.581ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_cockroachdb_734a7da0-4664-48e8-b4a5-4a758b3f3876
15902024-08-14T22:30:28.581ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15912024-08-14T22:30:28.581ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15922024-08-14T22:30:28.582ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 22783560 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45096 req_id = 1343798b-318c-435e-8de7-4d5fdc8b7509 response_code = 204 uri = /omicron-zones
15932024-08-14T22:30:28.614ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54133
15942024-08-14T22:30:28.644ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3598
15952024-08-14T22:30:31.923ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3616
15962024-08-14T22:30:48.961ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3639
15972024-08-14T22:30:48.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 20346976 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:54133 req_id = 6581a9d9-72d9-47d7-ae80-e3540b24dd7e response_code = 204 uri = /cockroachdb
15982024-08-14T22:30:48.996ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: e319def5-0926-47c0-a571-e5e4b12c3c0d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 848ae27a-190f-4269-b82e-d54727102895, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f39622fa-bcc3-48cb-a68e-4adcea3ab653, kind: Service { id: 848ae27a-190f-4269-b82e-d54727102895 }, name: Name("ntp-848ae27a-190f-4269-b82e-d54727102895"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 096736ee-173c-4ab4-9218-85c2f4818050, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }, OmicronZoneConfig { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: b91e483c-c47c-4daa-a991-9e09554a7f98, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: de3c2053-cc1f-405f-977b-4cd7e79352bb, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 540301cc-bc41-4aa5-bb02-c6e25e1586b6, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 22b9691e-69fc-43d5-aa99-8a200c509905, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: 89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 51525209-f895-4011-a1fd-ddf7aaca081e, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: dc3f737d-b96f-4263-a212-59b00c5761a0, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 33ee4877-2136-4c89-b2d5-e9369e322337, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External } } } }, OmicronZoneConfig { id: 98df5eef-0a91-403e-9767-d03c82c93fc1, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External } } } }, OmicronZoneConfig { id: 89a308c2-7d36-4fd9-ba13-8685ba41ff64, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External } } } }, OmicronZoneConfig { id: 55f36055-586f-4a47-aacf-98b84018407c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 37279933-d27e-4265-9e70-d58d0575c79d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External } } } }, OmicronZoneConfig { id: bf4afffd-0248-4ed1-8c15-3ab55a5408c9, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool), kind: External } } } }, OmicronZoneConfig { id: eda2942d-b7e5-43c9-9a26-afd1ae8dfd53, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External } } } }, OmicronZoneConfig { id: 36352d33-9aad-4710-88fe-5c5851ee3285, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool), kind: External } } } }, OmicronZoneConfig { id: 30a7d6a1-1b15-49b5-ae95-f9abb51135aa, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
15992024-08-14T22:30:48.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64240
16002024-08-14T22:30:48.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16012024-08-14T22:30:48.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f6dfbfea-8bf4-4493-96d9-9d1b04d801d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16022024-08-14T22:30:49.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16032024-08-14T22:30:49.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e319def5-0926-47c0-a571-e5e4b12c3c0d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16042024-08-14T22:30:49.043ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16052024-08-14T22:30:49.043ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0895b1cd-15b7-45f6-983c-9e8b876b7c53, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16062024-08-14T22:30:49.064ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16072024-08-14T22:30:49.064ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c7a204e7-7a9e-433d-a12b-a29fd64ac00e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16082024-08-14T22:30:49.081ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16092024-08-14T22:30:49.081ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: e6f7e4b6-1ff5-456a-a3aa-13e35b2fda8b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16102024-08-14T22:30:49.100ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16112024-08-14T22:30:49.100ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4f9822a7-fb64-419c-8824-2c8538ed9a6a, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16122024-08-14T22:30:49.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16132024-08-14T22:30:49.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 096736ee-173c-4ab4-9218-85c2f4818050, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16142024-08-14T22:30:49.138ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16152024-08-14T22:30:49.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 734a7da0-4664-48e8-b4a5-4a758b3f3876, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16162024-08-14T22:30:49.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f90e9b4-6395-4b60-a377-b8ab146c790e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16172024-08-14T22:30:49.158ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 6f90e9b4-6395-4b60-a377-b8ab146c790e, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16182024-08-14T22:30:49.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2bd372ca-052a-4891-8e18-1e89afe09ee2, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16192024-08-14T22:30:49.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 2bd372ca-052a-4891-8e18-1e89afe09ee2, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16202024-08-14T22:30:49.393ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22b9691e-69fc-43d5-aa99-8a200c509905, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
16212024-08-14T22:30:49.393ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 22b9691e-69fc-43d5-aa99-8a200c509905, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16222024-08-14T22:30:49.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ee4877-2136-4c89-b2d5-e9369e322337, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16232024-08-14T22:30:49.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 33ee4877-2136-4c89-b2d5-e9369e322337, dataset_name: DatasetName { pool_name: ZpoolName { id: 17668df1-993b-458e-b1cf-310a88477d11 (zpool), kind: External }, kind: Crucible }, responder: ... }
16242024-08-14T22:30:49.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98df5eef-0a91-403e-9767-d03c82c93fc1, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16252024-08-14T22:30:49.634ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 98df5eef-0a91-403e-9767-d03c82c93fc1, dataset_name: DatasetName { pool_name: ZpoolName { id: 7f6d9db3-762e-4798-98cd-bb00d0a3b252 (zpool), kind: External }, kind: Crucible }, responder: ... }
16262024-08-14T22:30:49.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 89a308c2-7d36-4fd9-ba13-8685ba41ff64, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16272024-08-14T22:30:49.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 89a308c2-7d36-4fd9-ba13-8685ba41ff64, dataset_name: DatasetName { pool_name: ZpoolName { id: 1bd3ff04-304c-4362-b41f-8347b5dd9341 (zpool), kind: External }, kind: Crucible }, responder: ... }
16282024-08-14T22:30:49.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55f36055-586f-4a47-aacf-98b84018407c, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16292024-08-14T22:30:49.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 55f36055-586f-4a47-aacf-98b84018407c, dataset_name: DatasetName { pool_name: ZpoolName { id: db606de8-ae75-4597-bde9-3f1234d88ff3 (zpool), kind: External }, kind: Crucible }, responder: ... }
16302024-08-14T22:30:49.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 37279933-d27e-4265-9e70-d58d0575c79d, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16312024-08-14T22:30:49.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 37279933-d27e-4265-9e70-d58d0575c79d, dataset_name: DatasetName { pool_name: ZpoolName { id: 20226c3a-235f-4118-b164-e14967848f5c (zpool), kind: External }, kind: Crucible }, responder: ... }
16322024-08-14T22:30:50.087ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bf4afffd-0248-4ed1-8c15-3ab55a5408c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16332024-08-14T22:30:50.087ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: bf4afffd-0248-4ed1-8c15-3ab55a5408c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 2450844a-e7c2-49d0-8dc6-d59793fa3c37 (zpool), kind: External }, kind: Crucible }, responder: ... }
16342024-08-14T22:30:50.217ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eda2942d-b7e5-43c9-9a26-afd1ae8dfd53, dataset_name: DatasetName { pool_name: ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16352024-08-14T22:30:50.218ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eda2942d-b7e5-43c9-9a26-afd1ae8dfd53, dataset_name: DatasetName { pool_name: ZpoolName { id: a4af3495-6c0f-4fe3-b8a0-503d2a880027 (zpool), kind: External }, kind: Crucible }, responder: ... }
16362024-08-14T22:30:50.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 36352d33-9aad-4710-88fe-5c5851ee3285, dataset_name: DatasetName { pool_name: ZpoolName { id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16372024-08-14T22:30:50.316ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 36352d33-9aad-4710-88fe-5c5851ee3285, dataset_name: DatasetName { pool_name: ZpoolName { id: 0faf5893-5a63-4906-a7f5-cb36c6c1353b (zpool), kind: External }, kind: Crucible }, responder: ... }
16382024-08-14T22:30:50.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 30a7d6a1-1b15-49b5-ae95-f9abb51135aa, dataset_name: DatasetName { pool_name: ZpoolName { id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16392024-08-14T22:30:50.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 30a7d6a1-1b15-49b5-ae95-f9abb51135aa, dataset_name: DatasetName { pool_name: ZpoolName { id: 126123c3-ef9f-486e-8f31-7272296cc4d8 (zpool), kind: External }, kind: Crucible }, responder: ... }
16402024-08-14T22:30:50.573ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16412024-08-14T22:30:50.573ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
16422024-08-14T22:30:50.573ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
16432024-08-14T22:30:50.573ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16442024-08-14T22:30:50.700ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
16452024-08-14T22:30:50.739ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
16462024-08-14T22:30:50.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16472024-08-14T22:30:50.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16482024-08-14T22:30:50.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16492024-08-14T22:30:50.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16502024-08-14T22:30:50.751ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
16512024-08-14T22:30:50.776ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16522024-08-14T22:30:50.776ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16532024-08-14T22:30:50.776ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16542024-08-14T22:30:50.910ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
16552024-08-14T22:30:50.910ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16562024-08-14T22:30:50.987ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
16572024-08-14T22:30:50.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16582024-08-14T22:30:50.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16592024-08-14T22:30:50.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16602024-08-14T22:30:50.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16612024-08-14T22:30:50.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16622024-08-14T22:30:50.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16632024-08-14T22:30:50.987ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0 file = illumos-utils/src/zone.rs:316
16642024-08-14T22:30:50.992ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337 file = illumos-utils/src/zone.rs:316
16652024-08-14T22:30:50.996ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64 file = illumos-utils/src/zone.rs:316
16662024-08-14T22:30:51.001ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c file = illumos-utils/src/zone.rs:316
16672024-08-14T22:30:51.007ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e file = illumos-utils/src/zone.rs:316
16682024-08-14T22:30:51.012ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d file = illumos-utils/src/zone.rs:316
16692024-08-14T22:30:51.017ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9 file = illumos-utils/src/zone.rs:316
16702024-08-14T22:30:51.156ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0 file = illumos-utils/src/zone.rs:354
16712024-08-14T22:30:51.161ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337 file = illumos-utils/src/zone.rs:354
16722024-08-14T22:30:51.166ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64 file = illumos-utils/src/zone.rs:354
16732024-08-14T22:30:51.171ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c file = illumos-utils/src/zone.rs:354
16742024-08-14T22:30:51.177ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e file = illumos-utils/src/zone.rs:354
16752024-08-14T22:30:51.183ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d file = illumos-utils/src/zone.rs:354
16762024-08-14T22:30:51.188ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9 file = illumos-utils/src/zone.rs:354
16772024-08-14T22:30:51.194ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf file = illumos-utils/src/zone.rs:316
16782024-08-14T22:30:51.201ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53 file = illumos-utils/src/zone.rs:316
16792024-08-14T22:30:51.207ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905 file = illumos-utils/src/zone.rs:316
16802024-08-14T22:30:51.213ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285 file = illumos-utils/src/zone.rs:316
16812024-08-14T22:30:51.220ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa file = illumos-utils/src/zone.rs:316
16822024-08-14T22:30:51.227ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1 file = illumos-utils/src/zone.rs:316
16832024-08-14T22:30:51.233ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6 file = illumos-utils/src/zone.rs:316
16842024-08-14T22:30:51.241ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf file = illumos-utils/src/zone.rs:354
16852024-08-14T22:30:51.248ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53 file = illumos-utils/src/zone.rs:354
16862024-08-14T22:30:51.255ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905 file = illumos-utils/src/zone.rs:354
16872024-08-14T22:30:51.262ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285 file = illumos-utils/src/zone.rs:354
16882024-08-14T22:30:51.271ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa file = illumos-utils/src/zone.rs:354
16892024-08-14T22:30:51.278ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1 file = illumos-utils/src/zone.rs:354
16902024-08-14T22:30:51.286ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6 file = illumos-utils/src/zone.rs:354
16912024-08-14T22:30:51.422ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
16922024-08-14T22:30:51.537ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
16932024-08-14T22:30:51.640ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
16942024-08-14T22:30:51.759ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
16952024-08-14T22:30:51.831ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
16962024-08-14T22:30:51.834ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
16972024-08-14T22:30:51.834ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
16982024-08-14T22:30:51.900ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 186, 135]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
16992024-08-14T22:30:51.900ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
17002024-08-14T22:30:51.902ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
17012024-08-14T22:30:51.902ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
17022024-08-14T22:30:51.963ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 151, 107]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
17032024-08-14T22:30:51.963ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
17042024-08-14T22:30:51.965ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
17052024-08-14T22:30:51.965ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
17062024-08-14T22:30:52.025ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 137, 242]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
17072024-08-14T22:30:52.025ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
17082024-08-14T22:30:52.027ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
17092024-08-14T22:30:52.027ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
17102024-08-14T22:30:52.089ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 139, 92]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
17112024-08-14T22:30:52.089ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
17122024-08-14T22:30:52.091ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
17132024-08-14T22:30:52.091ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
17142024-08-14T22:30:52.147ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 203, 96]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
17152024-08-14T22:30:52.147ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
17162024-08-14T22:30:52.156ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
17172024-08-14T22:30:52.156ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
17182024-08-14T22:30:52.157ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
17192024-08-14T22:30:52.157ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
17202024-08-14T22:30:52.157ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
17212024-08-14T22:30:52.158ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
17222024-08-14T22:30:52.158ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
17232024-08-14T22:30:52.158ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
17242024-08-14T22:30:52.158ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
17252024-08-14T22:30:52.158ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
17262024-08-14T22:30:52.175ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
17272024-08-14T22:30:52.177ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17282024-08-14T22:30:52.225ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
17292024-08-14T22:30:52.230ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
17302024-08-14T22:30:52.234ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17312024-08-14T22:30:52.234ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17322024-08-14T22:30:52.234ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
17332024-08-14T22:30:52.238ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
17342024-08-14T22:30:52.241ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17352024-08-14T22:30:52.241ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17362024-08-14T22:30:52.507ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2 file = illumos-utils/src/zone.rs:316
17372024-08-14T22:30:52.514ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98 file = illumos-utils/src/zone.rs:316
17382024-08-14T22:30:52.521ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf file = illumos-utils/src/zone.rs:316
17392024-08-14T22:30:52.528ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e file = illumos-utils/src/zone.rs:316
17402024-08-14T22:30:52.535ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb file = illumos-utils/src/zone.rs:316
17412024-08-14T22:30:52.542ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2 file = illumos-utils/src/zone.rs:354
17422024-08-14T22:30:52.550ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98 file = illumos-utils/src/zone.rs:354
17432024-08-14T22:30:52.558ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf file = illumos-utils/src/zone.rs:354
17442024-08-14T22:30:52.565ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e file = illumos-utils/src/zone.rs:354
17452024-08-14T22:30:52.570ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb file = illumos-utils/src/zone.rs:354
17462024-08-14T22:30:53.743ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17472024-08-14T22:30:53.743ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
17482024-08-14T22:30:53.784ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17492024-08-14T22:30:53.784ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
17502024-08-14T22:30:53.798ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17512024-08-14T22:30:53.798ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
17522024-08-14T22:30:53.892ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41263
17532024-08-14T22:30:53.895ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 2404 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41263 req_id = 1a9298e6-dcab-415b-9774-23200fc662c6 response_code = 200 uri = /baseboard
17542024-08-14T22:30:53.935ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_1bd3ff04-304c-4362-b41f-8347b5dd9341/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='197994ce-59a7-4c4f-af81-b29583ad0960'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17552024-08-14T22:30:53.935ZINFOSledAgent (ServiceManager): Booting oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
17562024-08-14T22:30:54.004ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a4af3495-6c0f-4fe3-b8a0-503d2a880027/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='f27398f2-aece-48d8-90f2-25cc0da3f3b8'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17572024-08-14T22:30:54.004ZINFOSledAgent (ServiceManager): Booting oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
17582024-08-14T22:30:54.012ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_20226c3a-235f-4118-b164-e14967848f5c/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='8c294397-7ad3-4d60-91fa-026822ea2cd6'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17592024-08-14T22:30:54.012ZINFOSledAgent (ServiceManager): Booting oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
17602024-08-14T22:30:54.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 57.836816ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
17612024-08-14T22:30:54.029ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7f6d9db3-762e-4798-98cd-bb00d0a3b252/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='1cea2a95-21f3-40d2-aafa-ccce4cdb6b93'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17622024-08-14T22:30:54.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 65.492317ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
17632024-08-14T22:30:54.038ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_17668df1-993b-458e-b1cf-310a88477d11/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='51e5ddba-78e5-4a18-ba03-b7b7270a742a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17642024-08-14T22:30:54.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 56.410426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
17652024-08-14T22:30:54.048ZINFOSledAgent (ServiceManager): Booting oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
17662024-08-14T22:30:54.054ZINFOSledAgent (ServiceManager): Booting oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
17672024-08-14T22:30:54.062ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_2450844a-e7c2-49d0-8dc6-d59793fa3c37/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='98804976-ea5a-4e84-a0c6-6d28cb18c6da'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17682024-08-14T22:30:54.062ZINFOSledAgent (ServiceManager): Booting oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
17692024-08-14T22:30:54.072ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='540301cc-bc41-4aa5-bb02-c6e25e1586b6'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17702024-08-14T22:30:54.072ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
17712024-08-14T22:30:54.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 111.907108ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
17722024-08-14T22:30:54.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 124.203022ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
17732024-08-14T22:30:54.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 96.098725ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
17742024-08-14T22:30:54.165ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_db606de8-ae75-4597-bde9-3f1234d88ff3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d9f12409-ad94-40ea-928a-20cf524b7249'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17752024-08-14T22:30:54.165ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_126123c3-ef9f-486e-8f31-7272296cc4d8/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='692a9cdc-e0b6-4106-8cb3-e5e762d42ece'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17762024-08-14T22:30:54.165ZINFOSledAgent (ServiceManager): Booting oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
17772024-08-14T22:30:54.190ZINFOSledAgent (ServiceManager): Booting oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
17782024-08-14T22:30:54.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 286.870695ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
17792024-08-14T22:30:54.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 55.501943ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
17802024-08-14T22:30:54.321ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_0faf5893-5a63-4906-a7f5-cb36c6c1353b/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='e6c8cbe0-f13e-41f9-a0b2-ce3d42b53863'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17812024-08-14T22:30:54.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 193.763961ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
17822024-08-14T22:30:54.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 98.144391ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
17832024-08-14T22:30:54.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 123.538058ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
17842024-08-14T22:30:54.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 46.684937ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
17852024-08-14T22:30:54.808ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 544.284395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
17862024-08-14T22:30:54.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 31.867128ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
17872024-08-14T22:30:54.879ZINFOSledAgent (ServiceManager): Booting oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
17882024-08-14T22:30:54.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 67.102459ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
17892024-08-14T22:30:54.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 73.550094ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
17902024-08-14T22:30:58.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 502.084418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
17912024-08-14T22:30:59.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 50.815836ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
17922024-08-14T22:30:59.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 297.090355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
17932024-08-14T22:30:59.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 59.358085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
17942024-08-14T22:30:59.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 413.881164ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
17952024-08-14T22:30:59.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 143.097745ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
17962024-08-14T22:30:59.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 62.028437ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
17972024-08-14T22:30:59.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 607.047105ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
17982024-08-14T22:30:59.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 120.20044ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
17992024-08-14T22:31:00.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 28.907713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18002024-08-14T22:31:00.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 116.609094ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
18012024-08-14T22:31:00.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 139.454475ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
18022024-08-14T22:31:00.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 583.847995ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
18032024-08-14T22:31:00.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 87.087462ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
18042024-08-14T22:31:00.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 432.724232ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
18052024-08-14T22:31:00.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 73.792413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
18062024-08-14T22:31:00.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 729.680587ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
18072024-08-14T22:31:00.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 164.160933ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
18082024-08-14T22:31:00.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 147.361571ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18092024-08-14T22:31:00.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 586.787049ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
18102024-08-14T22:31:00.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 271.449458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
18112024-08-14T22:31:00.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 46.547168ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18122024-08-14T22:31:00.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 82.755818ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18132024-08-14T22:31:00.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 150.713655ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
18142024-08-14T22:31:00.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 273.018709ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
18152024-08-14T22:31:00.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 106.906092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
18162024-08-14T22:31:00.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 1.080318424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
18172024-08-14T22:31:00.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 164.941958ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
18182024-08-14T22:31:00.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 516.411528ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
18192024-08-14T22:31:00.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 249.046815ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18202024-08-14T22:31:00.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 218.5092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
18212024-08-14T22:31:00.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 84.602472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18222024-08-14T22:31:00.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 181.805151ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18232024-08-14T22:31:00.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 470.049127ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
18242024-08-14T22:31:00.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 1.345281738s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
18252024-08-14T22:31:00.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 448.565346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
18262024-08-14T22:31:00.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 292.790303ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
18272024-08-14T22:31:00.937ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 178.782864ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18282024-08-14T22:31:00.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 441.84453ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
18292024-08-14T22:31:01.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 924.763177ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
18302024-08-14T22:31:01.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 687.902309ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
18312024-08-14T22:31:01.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 500.549239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
18322024-08-14T22:31:01.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 391.504355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18332024-08-14T22:31:01.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 500.736043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18342024-08-14T22:31:01.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 415.529338ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18352024-08-14T22:31:01.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.11205834s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
18362024-08-14T22:31:01.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 441.511622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
18372024-08-14T22:31:01.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 1.068336057s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
18382024-08-14T22:31:01.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 951.147328ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
18392024-08-14T22:31:01.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 832.154015ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
18402024-08-14T22:31:01.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 694.072888ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18412024-08-14T22:31:01.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 508.150419ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18422024-08-14T22:31:01.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.450181424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
18432024-08-14T22:31:01.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 709.993641ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18442024-08-14T22:31:01.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.159744059s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
18452024-08-14T22:31:01.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 742.802108ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
18462024-08-14T22:31:01.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 1.239327238s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
18472024-08-14T22:31:01.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 1.342433754s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
18482024-08-14T22:31:02.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 567.637335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18492024-08-14T22:31:02.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 868.287935ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18502024-08-14T22:31:02.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 1.140456777s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
18512024-08-14T22:31:02.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.178940086s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
18522024-08-14T22:31:02.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 529.090365ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18532024-08-14T22:31:02.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.16883342s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
18542024-08-14T22:31:02.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 951.010705ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
18552024-08-14T22:31:02.413ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18562024-08-14T22:31:02.415ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
18572024-08-14T22:31:02.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 1.293340141s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
18582024-08-14T22:31:02.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 1.249839111s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
18592024-08-14T22:31:02.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 668.93883ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18602024-08-14T22:31:02.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.028354321s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18612024-08-14T22:31:02.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 54.888184ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
18622024-08-14T22:31:03.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 556.650698ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
18632024-08-14T22:31:03.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 740.091023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
18642024-08-14T22:31:03.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 810.24599ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18652024-08-14T22:31:03.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 99.885819ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
18662024-08-14T22:31:03.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 699.416627ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
18672024-08-14T22:31:03.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 161.20684ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
18682024-08-14T22:31:03.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 840.893676ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
18692024-08-14T22:31:03.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 726.219706ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
18702024-08-14T22:31:03.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 927.129369ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18712024-08-14T22:31:03.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 722.044445ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
18722024-08-14T22:31:03.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 402.330833ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
18732024-08-14T22:31:03.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 892.750676ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
18742024-08-14T22:31:03.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.327603296s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
18752024-08-14T22:31:03.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 590.277049ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
18762024-08-14T22:31:03.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 514.153287ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
18772024-08-14T22:31:03.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 582.605048ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
18782024-08-14T22:31:03.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 476.994932ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
18792024-08-14T22:31:03.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.181582469s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
18802024-08-14T22:31:03.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 1.027721991s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
18812024-08-14T22:31:03.925ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18822024-08-14T22:31:03.925ZINFOSledAgent (ServiceManager): Booting oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
18832024-08-14T22:31:03.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 616.73027ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18842024-08-14T22:31:03.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 670.974326ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18852024-08-14T22:31:04.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.032217799s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
18862024-08-14T22:31:04.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 787.038141ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
18872024-08-14T22:31:04.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 734.706722ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
18882024-08-14T22:31:04.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.156266316s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
18892024-08-14T22:31:04.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 45.665429ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
18902024-08-14T22:31:04.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 1.233034335s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
18912024-08-14T22:31:04.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 511.626037ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
18922024-08-14T22:31:04.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.31254383s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
18932024-08-14T22:31:04.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 133.470634ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
18942024-08-14T22:31:04.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.338659881s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
18952024-08-14T22:31:04.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 759.784444ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
18962024-08-14T22:31:04.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 237.333688ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
18972024-08-14T22:31:04.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.113111238s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
18982024-08-14T22:31:04.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 1.426815938s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
18992024-08-14T22:31:04.750ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19002024-08-14T22:31:04.750ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19012024-08-14T22:31:04.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 438.940013ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
19022024-08-14T22:31:04.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.473215728s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
19032024-08-14T22:31:04.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.20293734s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
19042024-08-14T22:31:04.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 1.387606653s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
19052024-08-14T22:31:04.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 861.191137ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
19062024-08-14T22:31:04.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 1.310196852s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
19072024-08-14T22:31:05.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 54.15623ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19082024-08-14T22:31:05.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.088743305s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
19092024-08-14T22:31:05.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 112.326744ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19102024-08-14T22:31:05.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.336993035s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
19112024-08-14T22:31:05.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 1.328025354s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
19122024-08-14T22:31:05.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 772.971434ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
19132024-08-14T22:31:05.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 196.68357ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19142024-08-14T22:31:05.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.328817658s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
19152024-08-14T22:31:05.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 441.796141ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19162024-08-14T22:31:05.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 1.042602248s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
19172024-08-14T22:31:05.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.179385306s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
19182024-08-14T22:31:05.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.24358069s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
19192024-08-14T22:31:05.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.109256662s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
19202024-08-14T22:31:05.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 1.088203989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
19212024-08-14T22:31:05.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.094804084s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19222024-08-14T22:31:06.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 879.868238ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
19232024-08-14T22:31:06.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 794.39075ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
19242024-08-14T22:31:06.109ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19252024-08-14T22:31:06.109ZINFOSledAgent (ServiceManager): Booting oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19262024-08-14T22:31:06.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.294780417s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
19272024-08-14T22:31:06.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.474111444s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
19282024-08-14T22:31:06.242ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19292024-08-14T22:31:06.242ZINFOSledAgent (ServiceManager): Booting oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19302024-08-14T22:31:06.250ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="clickhouse_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19312024-08-14T22:31:06.250ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19322024-08-14T22:31:06.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 733.954003ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
19332024-08-14T22:31:06.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 62.566131ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19342024-08-14T22:31:06.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 1.210529615s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
19352024-08-14T22:31:06.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 102.690587ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19362024-08-14T22:31:06.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.367382691s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
19372024-08-14T22:31:06.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 65.753435ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19382024-08-14T22:31:06.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.48646848s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
19392024-08-14T22:31:06.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 115.990782ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19402024-08-14T22:31:06.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 76.374531ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19412024-08-14T22:31:06.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 51.718368ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19422024-08-14T22:31:06.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 717.560922ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
19432024-08-14T22:31:06.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 114.50004ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19442024-08-14T22:31:06.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 102.669011ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19452024-08-14T22:31:06.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 824.847233ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
19462024-08-14T22:31:06.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 484.263199ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19472024-08-14T22:31:06.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 172.774408ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19482024-08-14T22:31:06.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 490.466907ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19492024-08-14T22:31:06.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 674.48307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
19502024-08-14T22:31:06.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.368581182s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
19512024-08-14T22:31:06.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 769.789218ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
19522024-08-14T22:31:07.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 1.44381529s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
19532024-08-14T22:31:07.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 699.393044ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
19542024-08-14T22:31:07.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 257.950686ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19552024-08-14T22:31:07.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 893.533255ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
19562024-08-14T22:31:07.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.287943938s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
19572024-08-14T22:31:07.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 947.452884ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
19582024-08-14T22:31:07.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 659.648864ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19592024-08-14T22:31:07.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 925.99087ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19602024-08-14T22:31:07.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 785.409092ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19612024-08-14T22:31:07.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 618.123596ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
19622024-08-14T22:31:07.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 846.87833ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19632024-08-14T22:31:07.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.15770438s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
19642024-08-14T22:31:07.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 1.12482001s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
19652024-08-14T22:31:07.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 817.291135ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
19662024-08-14T22:31:07.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 763.799083ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
19672024-08-14T22:31:07.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.311273747s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
19682024-08-14T22:31:07.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 694.564749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
19692024-08-14T22:31:07.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.346509651s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19702024-08-14T22:31:07.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 765.613655ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
19712024-08-14T22:31:07.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 661.093272ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
19722024-08-14T22:31:07.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 828.588607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
19732024-08-14T22:31:07.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 1.12133239s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
19742024-08-14T22:31:07.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 902.632152ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
19752024-08-14T22:31:08.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 1.407862817s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
19762024-08-14T22:31:08.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.066066937s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19772024-08-14T22:31:08.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 590.801987ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19782024-08-14T22:31:08.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 580.764021ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19792024-08-14T22:31:08.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.093126289s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
19802024-08-14T22:31:08.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.038434127s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
19812024-08-14T22:31:08.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 838.811803ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
19822024-08-14T22:31:08.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 539.709678ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
19832024-08-14T22:31:08.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.340902355s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
19842024-08-14T22:31:08.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 1.16944886s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
19852024-08-14T22:31:08.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 826.859981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
19862024-08-14T22:31:08.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.143790591s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
19872024-08-14T22:31:08.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 773.592593ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
19882024-08-14T22:31:08.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 812.14097ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
19892024-08-14T22:31:08.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.285458867s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
19902024-08-14T22:31:08.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 921.578177ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
19912024-08-14T22:31:08.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 510.970768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
19922024-08-14T22:31:08.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.485322674s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
19932024-08-14T22:31:08.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.475368004s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
19942024-08-14T22:31:09.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.435606757s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
19952024-08-14T22:31:09.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 787.274335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
19962024-08-14T22:31:09.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 624.704138ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
19972024-08-14T22:31:09.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.219649219s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
19982024-08-14T22:31:09.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 999.312697ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
19992024-08-14T22:31:09.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.238594333s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
20002024-08-14T22:31:09.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 955.569445ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
20012024-08-14T22:31:09.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 736.777351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
20022024-08-14T22:31:09.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.426716238s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
20032024-08-14T22:31:09.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.475004022s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
20042024-08-14T22:31:09.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.068954748s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
20052024-08-14T22:31:09.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 1.295121277s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
20062024-08-14T22:31:09.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 539.743112ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
20072024-08-14T22:31:09.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.060518411s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
20082024-08-14T22:31:09.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.462714081s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
20092024-08-14T22:31:09.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 922.735184ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
20102024-08-14T22:31:09.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.022005421s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
20112024-08-14T22:31:09.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 848.916006ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
20122024-08-14T22:31:09.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.413917018s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
20132024-08-14T22:31:10.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 665.742599ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
20142024-08-14T22:31:10.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 1.239549688s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
20152024-08-14T22:31:10.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 676.817206ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
20162024-08-14T22:31:10.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 1.167996003s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
20172024-08-14T22:31:10.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 839.091762ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
20182024-08-14T22:31:10.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 606.87422ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
20192024-08-14T22:31:10.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.098164896s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
20202024-08-14T22:31:10.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.349366743s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
20212024-08-14T22:31:10.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.492602803s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
20222024-08-14T22:31:10.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 697.360799ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
20232024-08-14T22:31:10.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.354734485s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
20242024-08-14T22:31:10.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 816.215197ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
20252024-08-14T22:31:10.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.245543622s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
20262024-08-14T22:31:10.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 533.613761ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
20272024-08-14T22:31:10.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 1.381196732s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
20282024-08-14T22:31:10.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 681.21266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
20292024-08-14T22:31:10.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.26862029s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
20302024-08-14T22:31:10.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.322178541s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
20312024-08-14T22:31:10.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 584.123817ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
20322024-08-14T22:31:11.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.339796213s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
20332024-08-14T22:31:11.184ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.253550012s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
20342024-08-14T22:31:11.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.005071523s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
20352024-08-14T22:31:11.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 941.935439ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
20362024-08-14T22:31:11.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 538.051339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
20372024-08-14T22:31:11.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.45076524s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
20382024-08-14T22:31:11.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 1.457407427s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
20392024-08-14T22:31:11.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 693.111458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
20402024-08-14T22:31:11.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 780.890935ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
20412024-08-14T22:31:11.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 589.129482ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
20422024-08-14T22:31:11.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.309637824s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
20432024-08-14T22:31:11.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 512.740318ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
20442024-08-14T22:31:11.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 639.710395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
20452024-08-14T22:31:11.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 630.105241ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
20462024-08-14T22:31:12.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.032776797s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
20472024-08-14T22:31:12.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.445582685s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
20482024-08-14T22:31:12.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.430934974s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
20492024-08-14T22:31:12.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 955.535114ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
20502024-08-14T22:31:12.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.303662114s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
20512024-08-14T22:31:12.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.083383621s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
20522024-08-14T22:31:12.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 652.567728ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
20532024-08-14T22:31:12.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 662.527227ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
20542024-08-14T22:31:12.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 943.831577ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
20552024-08-14T22:31:12.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 596.231437ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
20562024-08-14T22:31:12.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.028777939s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
20572024-08-14T22:31:12.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 1.406954244s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
20582024-08-14T22:31:12.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 573.518824ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
20592024-08-14T22:31:12.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 561.781702ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
20602024-08-14T22:31:12.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 879.136638ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
20612024-08-14T22:31:12.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 610.419622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
20622024-08-14T22:31:12.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 799.308435ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
20632024-08-14T22:31:12.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 1.136580234s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
20642024-08-14T22:31:12.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.244322838s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
20652024-08-14T22:31:12.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.324455366s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
20662024-08-14T22:31:12.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 573.954828ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
20672024-08-14T22:31:12.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 626.135554ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
20682024-08-14T22:31:12.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.021631384s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
20692024-08-14T22:31:13.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.484225304s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
20702024-08-14T22:31:13.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.221702931s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
20712024-08-14T22:31:13.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 526.339827ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
20722024-08-14T22:31:13.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 592.003162ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
20732024-08-14T22:31:13.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 654.879161ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
20742024-08-14T22:31:13.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.241143159s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
20752024-08-14T22:31:13.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 845.27851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
20762024-08-14T22:31:13.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 797.597085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
20772024-08-14T22:31:13.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.355305904s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
20782024-08-14T22:31:13.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 1.215892836s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
20792024-08-14T22:31:13.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 1.365868099s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
20802024-08-14T22:31:13.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.179543672s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
20812024-08-14T22:31:13.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 758.839176ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
20822024-08-14T22:31:13.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 633.921268ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
20832024-08-14T22:31:13.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.002740988s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
20842024-08-14T22:31:13.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0") failed: Property not found. retry in 1.449934968s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
20852024-08-14T22:31:13.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64") failed: Property not found. retry in 1.180030501s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
20862024-08-14T22:31:13.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.038983178s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
20872024-08-14T22:31:13.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 995.493943ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
20882024-08-14T22:31:14.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 941.202172ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
20892024-08-14T22:31:14.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 625.128957ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
20902024-08-14T22:31:14.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.236894668s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
20912024-08-14T22:31:14.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 705.348415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
20922024-08-14T22:31:14.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 915.004029ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
20932024-08-14T22:31:14.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 1.061729065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
20942024-08-14T22:31:14.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.458045576s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
20952024-08-14T22:31:14.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e") failed: Property not found. retry in 1.168305074s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
20962024-08-14T22:31:14.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 649.746901ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
20972024-08-14T22:31:14.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 662.501768ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
20982024-08-14T22:31:14.568ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 78.541917677s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
20992024-08-14T22:31:14.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.077685449s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21002024-08-14T22:31:14.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 686.054987ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21012024-08-14T22:31:14.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf") failed: Property not found. retry in 708.758661ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
21022024-08-14T22:31:14.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 860.110169ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21032024-08-14T22:31:14.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 607.011144ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
21042024-08-14T22:31:14.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d") failed: Property not found. retry in 805.928657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
21052024-08-14T22:31:14.874ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 811.619703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
21062024-08-14T22:31:14.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.393782118s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21072024-08-14T22:31:15.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.229350101s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
21082024-08-14T22:31:15.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 636.330412ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
21092024-08-14T22:31:15.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6") failed: Property not found. retry in 1.49026487s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
21102024-08-14T22:31:15.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.360028969s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21112024-08-14T22:31:15.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.263572342s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21122024-08-14T22:31:15.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 1.484706181s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21132024-08-14T22:31:15.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 535.994718ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21142024-08-14T22:31:15.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 966.129538ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
21152024-08-14T22:31:15.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 639.535524ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
21162024-08-14T22:31:15.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.006971606s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21172024-08-14T22:31:15.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 678.184119ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
21182024-08-14T22:31:15.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 953.249771ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
21192024-08-14T22:31:15.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 866.492954ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21202024-08-14T22:31:15.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.431516524s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21212024-08-14T22:31:15.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.236593941s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21222024-08-14T22:31:16.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 557.279001ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
21232024-08-14T22:31:16.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.084260906s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21242024-08-14T22:31:16.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.270875376s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
21252024-08-14T22:31:16.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.023951115s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
21262024-08-14T22:31:16.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.462166851s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
21272024-08-14T22:31:16.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 994.032134ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21282024-08-14T22:31:16.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.213136928s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21292024-08-14T22:31:16.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9") failed: Property not found. retry in 1.073152801s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
21302024-08-14T22:31:16.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 891.54953ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21312024-08-14T22:31:16.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 643.495323ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21322024-08-14T22:31:16.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 1.075403157s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
21332024-08-14T22:31:16.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 863.47897ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21342024-08-14T22:31:17.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 634.804594ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21352024-08-14T22:31:17.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 905.541087ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21362024-08-14T22:31:17.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.102945019s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21372024-08-14T22:31:17.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.277550644s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21382024-08-14T22:31:17.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 724.950544ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
21392024-08-14T22:31:17.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.031355543s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21402024-08-14T22:31:17.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 555.365972ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21412024-08-14T22:31:17.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 1.256619091s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
21422024-08-14T22:31:17.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 612.42982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21432024-08-14T22:31:17.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 695.828867ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21442024-08-14T22:31:17.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 503.996691ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
21452024-08-14T22:31:17.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.168337186s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21462024-08-14T22:31:17.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 708.177642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
21472024-08-14T22:31:18.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.004487355s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21482024-08-14T22:31:18.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 770.176792ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21492024-08-14T22:31:18.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 530.868485ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
21502024-08-14T22:31:18.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 924.14937ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
21512024-08-14T22:31:18.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 800.431568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21522024-08-14T22:31:18.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.246447677s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21532024-08-14T22:31:18.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 567.173804ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21542024-08-14T22:31:18.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 901.301736ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21552024-08-14T22:31:18.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53") failed: Property not found. retry in 1.037716857s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
21562024-08-14T22:31:18.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 887.678716ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
21572024-08-14T22:31:18.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 858.02792ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21582024-08-14T22:31:18.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 651.069366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
21592024-08-14T22:31:18.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.304403023s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21602024-08-14T22:31:19.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 583.63383ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21612024-08-14T22:31:19.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 976.124583ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21622024-08-14T22:31:19.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 651.301514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21632024-08-14T22:31:19.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 900.2676ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21642024-08-14T22:31:19.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1") failed: Property not found. retry in 851.706923ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
21652024-08-14T22:31:19.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.009563435s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21662024-08-14T22:31:19.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c") failed: Property not found. retry in 724.241805ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
21672024-08-14T22:31:19.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 978.097829ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21682024-08-14T22:31:19.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337") failed: Property not found. retry in 1.43908569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
21692024-08-14T22:31:19.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 972.024009ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21702024-08-14T22:31:19.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.150569672s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21712024-08-14T22:31:19.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 596.240041ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21722024-08-14T22:31:20.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 757.872638ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21732024-08-14T22:31:20.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.435652512s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21742024-08-14T22:31:20.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 772.542654ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21752024-08-14T22:31:20.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 805.568189ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21762024-08-14T22:31:20.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 594.572839ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21772024-08-14T22:31:20.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 1.154443214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21782024-08-14T22:31:20.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.385055046s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21792024-08-14T22:31:20.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 622.733825ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21802024-08-14T22:31:20.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 732.329001ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21812024-08-14T22:31:21.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.184220134s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21822024-08-14T22:31:21.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 911.633619ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21832024-08-14T22:31:21.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 880.243104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21842024-08-14T22:31:21.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.097607829s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21852024-08-14T22:31:21.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.425635927s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21862024-08-14T22:31:21.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.1849738s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21872024-08-14T22:31:21.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 818.73269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21882024-08-14T22:31:22.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.14993055s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21892024-08-14T22:31:22.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 870.072353ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21902024-08-14T22:31:22.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa") failed: Property not found. retry in 1.245713567s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
21912024-08-14T22:31:22.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.01740026s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21922024-08-14T22:31:22.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb") failed: Property not found. retry in 1.132711056s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
21932024-08-14T22:31:22.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285") failed: Property not found. retry in 790.616011ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
21942024-08-14T22:31:22.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2") failed: Property not found. retry in 1.024171297s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
21952024-08-14T22:31:22.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e") failed: Property not found. retry in 1.10747089s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
21962024-08-14T22:31:22.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905") failed: Property not found. retry in 1.413895769s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
21972024-08-14T22:31:23.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf") failed: Property not found. retry in 1.033106265s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
21982024-08-14T22:31:23.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98") failed: Property not found. retry in 1.273668393s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
21992024-08-14T22:31:23.866ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22002024-08-14T22:31:23.965ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56161
22012024-08-14T22:31:23.969ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 2318 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56161 req_id = af09a3c9-8c5f-4cb6-b726-fdcfc7efa6a4 response_code = 200 uri = /baseboard
22022024-08-14T22:31:24.483ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22032024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_nexus_b91e483c-c47c-4daa-a991-9e09554a7f98
22042024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_pantry_dc3f737d-b96f-4263-a212-59b00c5761a0
22052024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_33ee4877-2136-4c89-b2d5-e9369e322337
22062024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_89a308c2-7d36-4fd9-ba13-8685ba41ff64
22072024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_55f36055-586f-4a47-aacf-98b84018407c
22082024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_nexus_30bf6548-3cd1-41c3-9086-faa8ca9474cf
22092024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_external_dns_6f90e9b4-6395-4b60-a377-b8ab146c790e
22102024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_pantry_51525209-f895-4011-a1fd-ddf7aaca081e
22112024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_37279933-d27e-4265-9e70-d58d0575c79d
22122024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_bf4afffd-0248-4ed1-8c15-3ab55a5408c9
22132024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_pantry_89d00a5b-7bfb-4c5d-b35f-6708f98fe6bf
22142024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_nexus_de3c2053-cc1f-405f-977b-4cd7e79352bb
22152024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_eda2942d-b7e5-43c9-9a26-afd1ae8dfd53
22162024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_clickhouse_22b9691e-69fc-43d5-aa99-8a200c509905
22172024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_36352d33-9aad-4710-88fe-5c5851ee3285
22182024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_30a7d6a1-1b15-49b5-ae95-f9abb51135aa
22192024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_crucible_98df5eef-0a91-403e-9767-d03c82c93fc1
22202024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_oximeter_540301cc-bc41-4aa5-bb02-c6e25e1586b6
22212024-08-14T22:31:24.565ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3162 zone = oxz_external_dns_2bd372ca-052a-4891-8e18-1e89afe09ee2
22222024-08-14T22:31:24.566ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
22232024-08-14T22:31:24.568ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
22242024-08-14T22:31:24.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 35575649 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64240 req_id = 65bea112-8067-4b9d-ab61-d46e24a1fbf2 response_code = 204 uri = /omicron-zones
22252024-08-14T22:31:24.588ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1250
22262024-08-14T22:31:24.588ZINFOSledAgent (RSS): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
22272024-08-14T22:31:24.588ZINFOSledAgent (RSS): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
22282024-08-14T22:31:24.593ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:687
22292024-08-14T22:31:24.601ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:700
22302024-08-14T22:31:24.648ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:831
22312024-08-14T22:31:24.652ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22322024-08-14T22:31:24.722ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22332024-08-14T22:31:24.761ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22342024-08-14T22:31:24.873ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22352024-08-14T22:31:24.894ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22362024-08-14T22:31:25.110ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22372024-08-14T22:31:25.233ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22382024-08-14T22:31:25.446ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22392024-08-14T22:31:25.726ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22402024-08-14T22:31:25.876ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22412024-08-14T22:31:25.917ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22422024-08-14T22:31:26.131ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22432024-08-14T22:31:26.419ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22442024-08-14T22:31:26.717ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22452024-08-14T22:31:26.879ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22462024-08-14T22:31:27.068ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22472024-08-14T22:31:27.881ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22482024-08-14T22:31:27.964ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22492024-08-14T22:31:28.666ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22502024-08-14T22:31:28.717ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22512024-08-14T22:31:28.883ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22522024-08-14T22:31:29.885ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22532024-08-14T22:31:30.004ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22542024-08-14T22:31:30.716ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22552024-08-14T22:31:30.889ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22562024-08-14T22:31:30.977ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22572024-08-14T22:31:31.892ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22582024-08-14T22:31:31.919ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22592024-08-14T22:31:32.717ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22602024-08-14T22:31:32.895ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22612024-08-14T22:31:33.833ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22622024-08-14T22:31:33.898ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/aeb7e077-0751-417b-8b16-b62876b5611c): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:229 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22632024-08-14T22:31:34.997ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "7a6f1fe4-668f-44f3-811f-761a8b7ef0c9", "content-length": "173", "date": "Wed, 14 Aug 2024 22:31:34 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"a0b038fc-3fac-44f8-9f1a-4ef58ea17b4f\"", request_id: "7a6f1fe4-668f-44f3-811f-761a8b7ef0c9" } file = sled-agent/src/rack_setup/service.rs:893
22642024-08-14T22:31:36.777ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:574 new_generation = 1 old_generation = 1 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
22652024-08-14T22:31:36.939ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64603
22662024-08-14T22:31:36.941ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22672024-08-14T22:31:37.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 92974 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64603 req_id = e7d4a47a-07a9-4f1b-baf5-2f2299fa8746 response_code = 200 uri = /inventory
22682024-08-14T22:31:37.033ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22692024-08-14T22:31:37.033ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
22702024-08-14T22:31:37.035ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
22712024-08-14T22:31:37.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 2297 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64603 req_id = 9d04f3bc-27aa-4395-9310-cfb491a6d595 response_code = 200 uri = /omicron-zones
22722024-08-14T22:31:37.912ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64337
22732024-08-14T22:31:37.916ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22742024-08-14T22:31:37.916ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22752024-08-14T22:31:37.916ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22762024-08-14T22:31:37.916ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22772024-08-14T22:31:37.916ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22782024-08-14T22:31:37.916ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22792024-08-14T22:31:37.916ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
22802024-08-14T22:31:37.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 4653 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64337 req_id = 1011df56-9ab2-4c10-94c9-f96525628ca4 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22812024-08-14T22:31:37.918ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:903
22822024-08-14T22:31:38.162ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57488
22832024-08-14T22:31:38.162ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22842024-08-14T22:31:38.258ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 96183 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57488 req_id = 7032b904-433f-4420-be85-2b6819c94f02 response_code = 200 uri = /inventory
22852024-08-14T22:31:38.259ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22862024-08-14T22:31:38.259ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/config/omicron-zones.json file = common/src/ledger.rs:177
22872024-08-14T22:31:38.260ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/config/omicron-zones.json file = common/src/ledger.rs:177
22882024-08-14T22:31:38.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1670 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57488 req_id = 1df11856-d188-4207-b62a-67ca1c1d01a9 response_code = 200 uri = /omicron-zones
22892024-08-14T22:31:38.435ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51528
22902024-08-14T22:31:38.435ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22912024-08-14T22:31:38.435ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22922024-08-14T22:31:38.435ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22932024-08-14T22:31:38.435ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22942024-08-14T22:31:38.435ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22952024-08-14T22:31:38.436ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22962024-08-14T22:31:38.436ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
22972024-08-14T22:31:38.436ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 440 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51528 req_id = 0acca6b5-d8f7-4428-a654-e4b6bd1ff1bd response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22982024-08-14T22:31:38.477ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36371
22992024-08-14T22:31:38.478ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23002024-08-14T22:31:38.478ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23012024-08-14T22:31:38.478ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23022024-08-14T22:31:38.478ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23032024-08-14T22:31:38.478ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23042024-08-14T22:31:38.478ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23052024-08-14T22:31:38.478ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
23062024-08-14T22:31:38.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 754 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36371 req_id = 9dd4ac56-84c9-4368-b513-bb29ba57bb02 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23072024-08-14T22:31:38.480ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32920
23082024-08-14T22:31:38.481ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23092024-08-14T22:31:38.481ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23102024-08-14T22:31:38.481ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23112024-08-14T22:31:38.481ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23122024-08-14T22:31:38.481ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23132024-08-14T22:31:38.481ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23142024-08-14T22:31:38.481ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
23152024-08-14T22:31:38.481ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 969 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:32920 req_id = 0d0cc227-5949-4a37-8dec-0f383a3aaf38 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23162024-08-14T22:31:45.406ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42795
23172024-08-14T22:31:45.408ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1081 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42795 req_id = 22c660bb-3aea-432f-9fd9-c968c703aefd response_code = 200 uri = /vpc-routes
23182024-08-14T22:31:45.467ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 2064 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:42795 req_id = 5c7518cc-e5bb-4cbf-909f-44137c957e89 response_code = 204 uri = /vpc-routes
23192024-08-14T22:31:54.006ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62736
23202024-08-14T22:31:54.006ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62736 req_id = 440fc6ca-0eec-486f-82ec-895fb7634a04 response_code = 200 uri = /baseboard
23212024-08-14T22:32:04.900ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61930
23222024-08-14T22:32:04.900ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61930 req_id = c18d208e-5812-4b74-83ce-df52547f7336 response_code = 200 uri = /vpc-routes
23232024-08-14T22:32:04.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61930 req_id = 5dc3838b-f55c-4eb9-882c-f2ea6cbf1eb9 response_code = 204 uri = /vpc-routes
23242024-08-14T22:32:04.961ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42652
23252024-08-14T22:32:04.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42652 req_id = d5f3e10e-b2e3-4853-85f1-2c1748dc228c response_code = 200 uri = /v2p
23262024-08-14T22:32:05.108ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38440
23272024-08-14T22:32:05.108ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
23282024-08-14T22:32:05.170ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
23292024-08-14T22:32:05.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 89192 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38440 req_id = f3a3b66c-f6bb-4f1a-a302-473e3de633ad response_code = 204 uri = /switch-ports
23302024-08-14T22:32:05.199ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1534 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38440 req_id = 96e60fb3-45bd-42d7-8400-0241e8d2de95 response_code = 200 uri = /network-bootstore-config
23312024-08-14T22:32:05.219ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
23322024-08-14T22:32:05.219ZINFOSledAgent (bootstore): Writing ledger to /pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
23332024-08-14T22:32:05.220ZINFOSledAgent (bootstore): Writing ledger to /pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
23342024-08-14T22:32:05.221ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
23352024-08-14T22:32:05.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 4588 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38440 req_id = 2f145bd6-e226-46ac-b337-ce216eddfd49 response_code = 204 uri = /network-bootstore-config
23362024-08-14T22:32:05.439ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55146
23372024-08-14T22:32:05.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55146 req_id = fba3f5e8-f346-4ba7-80bb-55c0b52f53ab response_code = 200 uri = /vpc-routes
23382024-08-14T22:32:05.450ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 864 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55146 req_id = 71b1f636-cdb3-4a2e-8ac8-18cc280606a9 response_code = 204 uri = /vpc-routes
23392024-08-14T22:32:05.483ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63487
23402024-08-14T22:32:05.483ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63487 req_id = 840a9884-6f31-4a42-8eec-bbbc7cc1ecd9 response_code = 200 uri = /v2p
23412024-08-14T22:32:05.603ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35889
23422024-08-14T22:32:05.603ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 24 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35889 req_id = be130863-1dd4-4818-a733-07aef787f79d response_code = 200 uri = /vpc-routes
23432024-08-14T22:32:05.612ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35889 req_id = c6c577d5-094a-4730-830d-b1925696bd8f response_code = 204 uri = /vpc-routes
23442024-08-14T22:32:05.631ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63698
23452024-08-14T22:32:05.631ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63698 req_id = b1d7c603-9cc8-42d6-bf78-75ce53e9efb4 response_code = 200 uri = /v2p
23462024-08-14T22:32:05.682ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53161
23472024-08-14T22:32:05.682ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
23482024-08-14T22:32:05.707ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
23492024-08-14T22:32:05.729ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 46676 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53161 req_id = 0388e5cb-49c7-4b7b-b8c0-9092b1d7533e response_code = 204 uri = /switch-ports
23502024-08-14T22:32:05.729ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50734
23512024-08-14T22:32:05.729ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 237 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53161 req_id = 00a86afe-c80c-4076-a72d-d8d91eba3ab0 response_code = 200 uri = /network-bootstore-config
23522024-08-14T22:32:05.730ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
23532024-08-14T22:32:05.754ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
23542024-08-14T22:32:05.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 47238 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50734 req_id = 9f82865f-f4f1-4bcf-af10-6ac4affb440c response_code = 204 uri = /switch-ports
23552024-08-14T22:32:05.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50734 req_id = 42dda709-bc3e-407b-9071-7246df90c9ef response_code = 200 uri = /network-bootstore-config
23562024-08-14T22:32:24.040ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35310
23572024-08-14T22:32:24.040ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35310 req_id = 616c49b8-9c98-4d50-b19d-c6ae1ea56d5b response_code = 200 uri = /baseboard
23582024-08-14T22:32:34.990ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57169
23592024-08-14T22:32:34.991ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 735 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57169 req_id = ee84c1bc-d9f5-4104-839d-17ba10597125 response_code = 200 uri = /vpc-routes
23602024-08-14T22:32:34.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57169 req_id = 6b177280-5caf-42c3-9df6-c6193af3bd8f response_code = 204 uri = /vpc-routes
23612024-08-14T22:32:35.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51846
23622024-08-14T22:32:35.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 378 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51846 req_id = b2b9343e-0311-4704-af3c-f478fa23edd0 response_code = 200 uri = /v2p
23632024-08-14T22:32:35.337ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34864
23642024-08-14T22:32:35.337ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
23652024-08-14T22:32:35.365ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
23662024-08-14T22:32:35.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 56998 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34864 req_id = d3201796-bc15-439a-946b-546c22f58093 response_code = 204 uri = /switch-ports
23672024-08-14T22:32:35.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34864 req_id = e8331009-e71f-4f4e-b130-caae078c19e1 response_code = 200 uri = /network-bootstore-config
23682024-08-14T22:32:35.627ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64584
23692024-08-14T22:32:35.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64584 req_id = 31b9e77a-0a97-49a5-b22a-2187bd544e21 response_code = 200 uri = /v2p
23702024-08-14T22:32:35.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56122
23712024-08-14T22:32:35.694ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56122 req_id = 38606b4f-f512-4ed8-bcfe-2f6bc6f3ee40 response_code = 200 uri = /vpc-routes
23722024-08-14T22:32:35.702ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56122 req_id = b343ff2c-e65b-4d99-b180-505cc4a7dbd4 response_code = 204 uri = /vpc-routes
23732024-08-14T22:32:35.711ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59995
23742024-08-14T22:32:35.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 22 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59995 req_id = eb880c4a-ad28-417a-8077-b23f90ce53c2 response_code = 200 uri = /vpc-routes
23752024-08-14T22:32:35.719ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 845 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:59995 req_id = 9abca4ea-8112-41a4-a9bf-bce83a28c50c response_code = 204 uri = /vpc-routes
23762024-08-14T22:32:35.821ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33749
23772024-08-14T22:32:35.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33749 req_id = 026dd237-87ee-4b37-857a-26b660f33adb response_code = 200 uri = /v2p
23782024-08-14T22:32:35.928ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58476
23792024-08-14T22:32:35.928ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
23802024-08-14T22:32:35.953ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
23812024-08-14T22:32:35.975ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 47839 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58476 req_id = 108a5bfe-9496-41c8-aed2-df2f079e68a6 response_code = 204 uri = /switch-ports
23822024-08-14T22:32:35.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58476 req_id = 7d278ccf-2df6-499f-8ab5-0dae3939c5a6 response_code = 200 uri = /network-bootstore-config
23832024-08-14T22:32:36.009ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61186
23842024-08-14T22:32:36.009ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
23852024-08-14T22:32:36.033ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
23862024-08-14T22:32:36.055ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 46201 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:61186 req_id = 2a18791b-6412-4039-a73d-008cbf027b21 response_code = 204 uri = /switch-ports
23872024-08-14T22:32:36.055ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61186 req_id = 3e543c44-95d3-42d7-9b14-6c3c373c993c response_code = 200 uri = /network-bootstore-config
23882024-08-14T22:32:47.715ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
23892024-08-14T22:32:54.073ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:50174
23902024-08-14T22:32:54.073ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:50174 req_id = f6db4031-9d29-47b9-a701-caaf7b0f68f3 response_code = 200 uri = /baseboard
23912024-08-14T22:33:03.252ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:49293 remote_addr = [fd00:1122:3344:101::d]:48533 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
23922024-08-14T22:33:03.301ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49566 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = c12033fb-28ab-4239-9e62-48b125f18e36 response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
23932024-08-14T22:33:04.902ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58707
23942024-08-14T22:33:04.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58707 req_id = c773afbc-fd0c-4599-b541-c4d02174cc31 response_code = 200 uri = /vpc-routes
23952024-08-14T22:33:04.909ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58707 req_id = 97c1255f-a632-4a86-978c-eaada2e987d0 response_code = 204 uri = /vpc-routes
23962024-08-14T22:33:04.977ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64935
23972024-08-14T22:33:04.978ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64935 req_id = 2b32269b-a22f-47e1-9d79-ff49e25a5688 response_code = 200 uri = /v2p
23982024-08-14T22:33:05.056ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32885
23992024-08-14T22:33:05.056ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24002024-08-14T22:33:05.081ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24012024-08-14T22:33:05.105ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49414 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:32885 req_id = a6fcbf9b-92de-400f-a504-f7e128037cc6 response_code = 204 uri = /switch-ports
24022024-08-14T22:33:05.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32885 req_id = a44841ec-5262-4612-ae5c-f5d4410487b6 response_code = 200 uri = /network-bootstore-config
24032024-08-14T22:33:05.435ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54345
24042024-08-14T22:33:05.435ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54345 req_id = b0a273c7-82a7-4c80-a3cb-34487932f56e response_code = 200 uri = /vpc-routes
24052024-08-14T22:33:05.444ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54345 req_id = 32d867d5-5543-474a-b867-ac724c30f5e9 response_code = 204 uri = /vpc-routes
24062024-08-14T22:33:05.512ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38061
24072024-08-14T22:33:05.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38061 req_id = d2480c87-927a-4c1f-906f-474748de0a9a response_code = 200 uri = /vpc-routes
24082024-08-14T22:33:05.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:38061 req_id = 225b5d78-bed9-42b5-8d5b-ec1c36944f2a response_code = 204 uri = /vpc-routes
24092024-08-14T22:33:05.558ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41617
24102024-08-14T22:33:05.558ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41617 req_id = 72ff659b-4e30-4a3e-b1a5-31329c80cfaa response_code = 200 uri = /v2p
24112024-08-14T22:33:05.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59044
24122024-08-14T22:33:05.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 718 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59044 req_id = 06a0df45-6629-4cb6-a54e-511846c6a4e3 response_code = 200 uri = /v2p
24132024-08-14T22:33:05.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52156
24142024-08-14T22:33:05.694ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24152024-08-14T22:33:05.721ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24162024-08-14T22:33:05.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 50604 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52156 req_id = 82c212a8-28e0-405b-a4cd-409103671eb1 response_code = 204 uri = /switch-ports
24172024-08-14T22:33:05.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52156 req_id = 1d5e3901-0364-4927-9fe9-dbd2ea548c94 response_code = 200 uri = /network-bootstore-config
24182024-08-14T22:33:05.746ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63728
24192024-08-14T22:33:05.747ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24202024-08-14T22:33:05.774ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24212024-08-14T22:33:05.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 50023 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63728 req_id = fab0ec21-c817-4d96-ba13-5d3b9ccb3966 response_code = 204 uri = /switch-ports
24222024-08-14T22:33:05.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63728 req_id = 1ab68839-0fad-4fc2-8237-9a64f3254b7a response_code = 200 uri = /network-bootstore-config
24232024-08-14T22:33:24.106ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51556
24242024-08-14T22:33:24.106ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 618 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51556 req_id = c11690b4-5010-45c9-9931-afe5d2bcc939 response_code = 200 uri = /baseboard
24252024-08-14T22:33:33.258ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 6640 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 7098ee65-f15a-4afa-a82b-6f6611973100 response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
24262024-08-14T22:33:35.084ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42680
24272024-08-14T22:33:35.085ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 894 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42680 req_id = 9d37c1bb-cc45-4d7e-8279-188e3c7f5ef2 response_code = 200 uri = /v2p
24282024-08-14T22:33:35.130ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52081
24292024-08-14T22:33:35.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52081 req_id = 9b156a36-2cfd-4960-af91-b2106cc3940d response_code = 200 uri = /vpc-routes
24302024-08-14T22:33:35.138ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:52081 req_id = 1ae1403c-04d6-4901-aefd-6a52802d11a9 response_code = 204 uri = /vpc-routes
24312024-08-14T22:33:35.439ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52944
24322024-08-14T22:33:35.439ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24332024-08-14T22:33:35.470ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24342024-08-14T22:33:35.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 57308 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52944 req_id = afb40e5e-ca2a-46c5-859e-5332269c9f0a response_code = 204 uri = /switch-ports
24352024-08-14T22:33:35.497ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 253 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52944 req_id = 48130ff5-8f71-4ef3-b3bc-367937ea341b response_code = 200 uri = /network-bootstore-config
24362024-08-14T22:33:35.548ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39049
24372024-08-14T22:33:35.548ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39049 req_id = f890c769-e351-4aa9-aa12-797c512f5bab response_code = 200 uri = /vpc-routes
24382024-08-14T22:33:35.555ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39049 req_id = a3ecf16f-1ee9-451e-a41c-3b2d03062155 response_code = 204 uri = /vpc-routes
24392024-08-14T22:33:35.635ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54872
24402024-08-14T22:33:35.635ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 22 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54872 req_id = 4181fbed-3fbd-4c08-b7a2-ba7e7de3a87e response_code = 200 uri = /vpc-routes
24412024-08-14T22:33:35.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54872 req_id = 2228b762-0efd-48a2-be92-82b07b62c145 response_code = 204 uri = /vpc-routes
24422024-08-14T22:33:35.696ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42151
24432024-08-14T22:33:35.697ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42151 req_id = c5269d30-46e5-4b1d-aa77-327423b65545 response_code = 200 uri = /v2p
24442024-08-14T22:33:35.776ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51366
24452024-08-14T22:33:35.776ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24462024-08-14T22:33:35.804ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24472024-08-14T22:33:35.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49746 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51366 req_id = fa76fd22-cbe0-4a24-965f-3a5e63b5007a response_code = 204 uri = /switch-ports
24482024-08-14T22:33:35.826ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37317
24492024-08-14T22:33:35.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 253 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51366 req_id = bfc7c4fa-1657-446e-8d91-91fa8d33e2ce response_code = 200 uri = /network-bootstore-config
24502024-08-14T22:33:35.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 245 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37317 req_id = 36ca6a3c-3ca7-43a9-9be1-7dd862f64607 response_code = 200 uri = /v2p
24512024-08-14T22:33:36.042ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60591
24522024-08-14T22:33:36.042ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24532024-08-14T22:33:36.070ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24542024-08-14T22:33:36.095ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 52948 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60591 req_id = da7dfd2e-c4c1-488b-a1ed-8b0239dd3eb3 response_code = 204 uri = /switch-ports
24552024-08-14T22:33:36.095ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60591 req_id = 0405995c-8bf1-4483-8113-5bf18abeba9a response_code = 200 uri = /network-bootstore-config
24562024-08-14T22:33:54.139ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33474
24572024-08-14T22:33:54.139ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33474 req_id = acc14042-2359-4a02-a191-05664ea8398c response_code = 200 uri = /baseboard
24582024-08-14T22:34:03.258ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 6830 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 103c64a9-413d-4feb-8dab-0131c044221a response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
24592024-08-14T22:34:05.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39338
24602024-08-14T22:34:05.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39338 req_id = 8644e6ee-de24-43f9-90d4-a752bf44d8ff response_code = 200 uri = /vpc-routes
24612024-08-14T22:34:05.037ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51471
24622024-08-14T22:34:05.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 613 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51471 req_id = fc194961-74ee-4e23-9f9d-d6d334ea7674 response_code = 200 uri = /v2p
24632024-08-14T22:34:05.041ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39338 req_id = efc64659-2bbf-400e-9468-81332b8ef50c response_code = 204 uri = /vpc-routes
24642024-08-14T22:34:05.143ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44498
24652024-08-14T22:34:05.143ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24662024-08-14T22:34:05.171ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24672024-08-14T22:34:05.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 51080 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44498 req_id = 6f698e32-d725-4f9f-8762-87ee88c822a9 response_code = 204 uri = /switch-ports
24682024-08-14T22:34:05.195ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44498 req_id = 4ed9a8f7-bfde-40c0-b9a7-9bc441d4dc33 response_code = 200 uri = /network-bootstore-config
24692024-08-14T22:34:05.428ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50307
24702024-08-14T22:34:05.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50307 req_id = a24b77c0-df84-4586-83dd-46ee2f9513b8 response_code = 200 uri = /vpc-routes
24712024-08-14T22:34:05.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50307 req_id = 00d3bc96-de8a-429f-9bf4-798ab3d7e4f9 response_code = 204 uri = /vpc-routes
24722024-08-14T22:34:05.481ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39125
24732024-08-14T22:34:05.481ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39125 req_id = c4f7b1f0-8791-48c5-85f7-5b5dff2427b1 response_code = 200 uri = /v2p
24742024-08-14T22:34:05.573ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53939
24752024-08-14T22:34:05.573ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53939 req_id = bb72cab9-74fd-473e-b338-416bbb73466e response_code = 200 uri = /v2p
24762024-08-14T22:34:05.587ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48624
24772024-08-14T22:34:05.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 845 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48624 req_id = ae1d94cb-8243-4989-9542-9976d37a8543 response_code = 200 uri = /vpc-routes
24782024-08-14T22:34:05.595ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48624 req_id = fce221ce-734a-4bfb-9afe-23720baba026 response_code = 204 uri = /vpc-routes
24792024-08-14T22:34:05.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51882
24802024-08-14T22:34:05.610ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24812024-08-14T22:34:05.638ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24822024-08-14T22:34:05.658ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48453 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51882 req_id = 07975f8a-5375-4592-90bd-a40ec78bf1a1 response_code = 204 uri = /switch-ports
24832024-08-14T22:34:05.658ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51882 req_id = 8a3ca150-f698-4271-b718-8b0a11b0f080 response_code = 200 uri = /network-bootstore-config
24842024-08-14T22:34:05.715ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40568
24852024-08-14T22:34:05.716ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24862024-08-14T22:34:05.741ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
24872024-08-14T22:34:05.764ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49391 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40568 req_id = 00d4a73a-46dc-4f53-bce1-061e8e192fd6 response_code = 204 uri = /switch-ports
24882024-08-14T22:34:05.765ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40568 req_id = 1e7ecfb6-d3f1-47ac-a3de-c6124affd6e5 response_code = 200 uri = /network-bootstore-config
24892024-08-14T22:34:24.172ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49439
24902024-08-14T22:34:24.172ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49439 req_id = 380ca3ba-9b8c-4c6a-8915-97535e5fbc1d response_code = 200 uri = /baseboard
24912024-08-14T22:34:33.259ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 6759 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 23fa6bab-8fb2-4ca8-b981-5824cb747df1 response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
24922024-08-14T22:34:35.141ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53301
24932024-08-14T22:34:35.141ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 194 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53301 req_id = 4e0928d6-eb77-43c7-9119-775cd7b81f10 response_code = 200 uri = /v2p
24942024-08-14T22:34:35.153ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49844
24952024-08-14T22:34:35.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 30 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49844 req_id = 1ecc1546-33ff-47ef-8f3f-0b2bda9fa4e4 response_code = 200 uri = /vpc-routes
24962024-08-14T22:34:35.159ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49844 req_id = eb4edb6e-7f2b-4ad5-b68a-e237455feebb response_code = 204 uri = /vpc-routes
24972024-08-14T22:34:35.474ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60315
24982024-08-14T22:34:35.474ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
24992024-08-14T22:34:35.501ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25002024-08-14T22:34:35.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 52816 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60315 req_id = d3086b6a-3363-4785-a559-44a8c4c87ed3 response_code = 204 uri = /switch-ports
25012024-08-14T22:34:35.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 202 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60315 req_id = 741e209f-7ff7-48dc-b083-9d8e785e7bb4 response_code = 200 uri = /network-bootstore-config
25022024-08-14T22:34:35.604ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48576
25032024-08-14T22:34:35.604ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48576 req_id = 01ca307d-ace2-4379-bf86-b28bd53f59d5 response_code = 200 uri = /v2p
25042024-08-14T22:34:35.639ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42377
25052024-08-14T22:34:35.639ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42377 req_id = 91bc0d69-5b3d-46b5-a50b-b5eda4c0d2f9 response_code = 200 uri = /vpc-routes
25062024-08-14T22:34:35.645ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42377 req_id = 9cafbb7f-b1ae-4460-ae57-acea4ef65d2f response_code = 204 uri = /vpc-routes
25072024-08-14T22:34:35.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44761
25082024-08-14T22:34:35.661ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44761 req_id = bbf90280-0973-4fe3-996a-d6854d2f07a1 response_code = 200 uri = /v2p
25092024-08-14T22:34:35.861ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47105
25102024-08-14T22:34:35.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47105 req_id = d9d6f54e-49eb-4726-aa3b-b2bd536de4d0 response_code = 200 uri = /vpc-routes
25112024-08-14T22:34:35.868ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47105 req_id = d745d5bc-aaa9-4fb0-bf7a-b26e5efc2916 response_code = 204 uri = /vpc-routes
25122024-08-14T22:34:35.955ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63936
25132024-08-14T22:34:35.955ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25142024-08-14T22:34:35.982ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25152024-08-14T22:34:36.007ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 51314 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63936 req_id = 0470d86a-df9b-44f3-8729-52197960edd7 response_code = 204 uri = /switch-ports
25162024-08-14T22:34:36.007ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63936 req_id = 04764c96-1f4d-4d35-b358-baf95cdd5efb response_code = 200 uri = /network-bootstore-config
25172024-08-14T22:34:36.081ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58307
25182024-08-14T22:34:36.082ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25192024-08-14T22:34:36.108ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25202024-08-14T22:34:36.132ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 50758 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58307 req_id = ac5b92c6-6562-4408-abc1-88087cdf2df0 response_code = 204 uri = /switch-ports
25212024-08-14T22:34:36.132ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58307 req_id = 392d0ce1-882f-4252-b6cd-309714e582be response_code = 200 uri = /network-bootstore-config
25222024-08-14T22:34:54.208ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41317
25232024-08-14T22:34:54.208ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41317 req_id = d4dda463-c4e7-43ea-888a-4d8dba3ab751 response_code = 200 uri = /baseboard
25242024-08-14T22:35:03.270ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 6108 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 271fd7b3-1fef-4995-a01a-d5999c53c77d response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
25252024-08-14T22:35:04.906ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47776
25262024-08-14T22:35:04.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 344 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47776 req_id = d31219ce-c524-42c9-a81e-f448ce133a95 response_code = 200 uri = /vpc-routes
25272024-08-14T22:35:04.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35356
25282024-08-14T22:35:04.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35356 req_id = dd7e9177-f912-4873-a1ac-53bbf5031e85 response_code = 200 uri = /v2p
25292024-08-14T22:35:04.913ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47776 req_id = 813b3238-de9e-4f0c-8f91-0149c989a41a response_code = 204 uri = /vpc-routes
25302024-08-14T22:35:05.149ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61938
25312024-08-14T22:35:05.149ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25322024-08-14T22:35:05.176ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25332024-08-14T22:35:05.199ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 50488 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61938 req_id = c132cf4a-254a-476e-b307-631e7221fdfc response_code = 204 uri = /switch-ports
25342024-08-14T22:35:05.202ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61938 req_id = bb178fa2-619e-495b-b1ef-63d6d05daa83 response_code = 200 uri = /network-bootstore-config
25352024-08-14T22:35:05.453ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51030
25362024-08-14T22:35:05.453ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51030 req_id = 19135182-24ae-4a32-b025-6bfcb9ad8eff response_code = 200 uri = /vpc-routes
25372024-08-14T22:35:05.461ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51030 req_id = 78d57273-51a1-42ef-a5e0-7eb88a8ea2e3 response_code = 204 uri = /vpc-routes
25382024-08-14T22:35:05.522ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52068
25392024-08-14T22:35:05.522ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52068 req_id = 651bed64-a49a-4823-ba95-5b26d7ba5e64 response_code = 200 uri = /vpc-routes
25402024-08-14T22:35:05.530ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 886 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52068 req_id = ab49a7c7-ea90-4e4a-a7b7-710359ee884a response_code = 204 uri = /vpc-routes
25412024-08-14T22:35:05.534ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61031
25422024-08-14T22:35:05.534ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61031 req_id = 27d9ddaa-adf0-4ce0-9667-6cab20b7a5b0 response_code = 200 uri = /v2p
25432024-08-14T22:35:05.556ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52411
25442024-08-14T22:35:05.556ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52411 req_id = 22b401b1-d57a-4da0-a3db-eb2c4d97128e response_code = 200 uri = /v2p
25452024-08-14T22:35:05.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34374
25462024-08-14T22:35:05.637ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25472024-08-14T22:35:05.664ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25482024-08-14T22:35:05.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 52304 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34374 req_id = 60dbdc9f-d9b3-44f3-9aca-40696d286560 response_code = 204 uri = /switch-ports
25492024-08-14T22:35:05.689ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38953
25502024-08-14T22:35:05.690ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25512024-08-14T22:35:05.694ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 4190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34374 req_id = 4d74a4e2-2cc2-4b3f-8de8-a510517c4fba response_code = 200 uri = /network-bootstore-config
25522024-08-14T22:35:05.714ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25532024-08-14T22:35:05.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 46173 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:38953 req_id = f0de41c0-998c-4061-8f37-428cfaf5c0c8 response_code = 204 uri = /switch-ports
25542024-08-14T22:35:05.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38953 req_id = 1da31505-d4a2-4653-801b-de695588320b response_code = 200 uri = /network-bootstore-config
25552024-08-14T22:35:24.241ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38204
25562024-08-14T22:35:24.242ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 976 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38204 req_id = a542434d-bdf4-4484-9af1-58c0904697d4 response_code = 200 uri = /baseboard
25572024-08-14T22:35:33.312ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 7126 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 953cb79c-b792-4af5-965f-bfa5b5263ac3 response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
25582024-08-14T22:35:34.916ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49275
25592024-08-14T22:35:34.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 665 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49275 req_id = e8434cf3-e904-4ef8-aef1-f0079cfde231 response_code = 200 uri = /vpc-routes
25602024-08-14T22:35:34.924ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49275 req_id = 87a39438-ca57-41dc-8e4a-324a90451722 response_code = 204 uri = /vpc-routes
25612024-08-14T22:35:35.003ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62649
25622024-08-14T22:35:35.003ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62649 req_id = f2bf2a8c-413e-4459-bb1a-bd34f9e78a10 response_code = 200 uri = /v2p
25632024-08-14T22:35:35.179ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38620
25642024-08-14T22:35:35.179ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25652024-08-14T22:35:35.203ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25662024-08-14T22:35:35.225ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 46123 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38620 req_id = 72d32af5-17c0-47df-a2ff-73cb7c8049f9 response_code = 204 uri = /switch-ports
25672024-08-14T22:35:35.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 613 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38620 req_id = d3c26d0c-737b-4a51-b75e-2d4b0088c81a response_code = 200 uri = /network-bootstore-config
25682024-08-14T22:35:35.551ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35673
25692024-08-14T22:35:35.552ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1196 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35673 req_id = 957d07d4-fe28-4e3b-b922-dad1c807439b response_code = 200 uri = /v2p
25702024-08-14T22:35:35.586ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60298
25712024-08-14T22:35:35.586ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60298 req_id = 7c43f7f9-0a03-4c28-b5f4-9d3e58ebf491 response_code = 200 uri = /vpc-routes
25722024-08-14T22:35:35.597ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1185 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60298 req_id = 94643f5a-4370-4c46-98d3-48b027abe7dd response_code = 204 uri = /vpc-routes
25732024-08-14T22:35:35.658ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33664
25742024-08-14T22:35:35.658ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33664 req_id = 23b31c56-70bc-4a7d-8843-359b4b1bc6f8 response_code = 200 uri = /vpc-routes
25752024-08-14T22:35:35.667ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 970 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33664 req_id = b1b29649-137d-40ec-868e-eafc2cd5d468 response_code = 204 uri = /vpc-routes
25762024-08-14T22:35:35.827ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52297
25772024-08-14T22:35:35.827ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52297 req_id = 6c8c843f-5040-46d0-96d7-f527ce7e9ec0 response_code = 200 uri = /v2p
25782024-08-14T22:35:35.929ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50811
25792024-08-14T22:35:35.929ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25802024-08-14T22:35:35.956ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25812024-08-14T22:35:35.977ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48511 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50811 req_id = e1f1a3d9-4a37-4f1c-ba50-1227ffd10c15 response_code = 204 uri = /switch-ports
25822024-08-14T22:35:35.977ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56750
25832024-08-14T22:35:35.977ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25842024-08-14T22:35:35.981ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 3248 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50811 req_id = d062ea57-4783-4a7e-8360-2713d4487f05 response_code = 200 uri = /network-bootstore-config
25852024-08-14T22:35:36.002ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25862024-08-14T22:35:36.024ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 47124 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:56750 req_id = ecc35773-cbda-4001-b4e1-3309c18942e8 response_code = 204 uri = /switch-ports
25872024-08-14T22:35:36.025ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56750 req_id = 72264e6c-45c5-453e-8a5c-0012c8178baf response_code = 200 uri = /network-bootstore-config
25882024-08-14T22:35:54.274ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42191
25892024-08-14T22:35:54.274ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42191 req_id = 733f07da-4b6c-4fbd-9914-5c303918718b response_code = 200 uri = /baseboard
25902024-08-14T22:36:03.313ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 6976 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 0ccfb27b-3942-4179-b968-91adfb6b9ed4 response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
25912024-08-14T22:36:04.891ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47505
25922024-08-14T22:36:04.891ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47505 req_id = bdfb9be8-b131-43f5-adc1-f8ce10cd064e response_code = 200 uri = /vpc-routes
25932024-08-14T22:36:04.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47505 req_id = 8775d280-85c7-4f7e-a29f-1fc747b78823 response_code = 204 uri = /vpc-routes
25942024-08-14T22:36:04.902ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40545
25952024-08-14T22:36:04.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40545 req_id = d5d8ae70-f31b-4c3d-ac27-b13dbe335fdb response_code = 200 uri = /v2p
25962024-08-14T22:36:05.053ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39830
25972024-08-14T22:36:05.053ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
25982024-08-14T22:36:05.078ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
25992024-08-14T22:36:05.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49688 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39830 req_id = 2e353caf-ce9e-4f8c-b7fc-f9f72f8eba8e response_code = 204 uri = /switch-ports
26002024-08-14T22:36:05.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39830 req_id = fcd6f8a8-84f9-41fb-83da-ad249c839961 response_code = 200 uri = /network-bootstore-config
26012024-08-14T22:36:05.475ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39107
26022024-08-14T22:36:05.475ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39107 req_id = 3e7a998c-b923-4ec7-876d-983e674576a4 response_code = 200 uri = /v2p
26032024-08-14T22:36:05.530ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49756
26042024-08-14T22:36:05.530ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49756 req_id = accafd60-3d14-49cf-a307-76ed49f30965 response_code = 200 uri = /vpc-routes
26052024-08-14T22:36:05.538ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49756 req_id = 4131f157-ae60-4044-adfd-d58ba6530222 response_code = 204 uri = /vpc-routes
26062024-08-14T22:36:05.540ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53319
26072024-08-14T22:36:05.541ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53319 req_id = 58dd00a6-2060-40cf-8efa-8b2c855449f4 response_code = 200 uri = /v2p
26082024-08-14T22:36:05.559ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59155
26092024-08-14T22:36:05.559ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 865 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59155 req_id = 621b395e-5166-4259-82ea-e1cc687dee36 response_code = 200 uri = /vpc-routes
26102024-08-14T22:36:05.566ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59155 req_id = e2bfcc45-7c06-408c-8be6-b890346dfd9c response_code = 204 uri = /vpc-routes
26112024-08-14T22:36:05.665ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36576
26122024-08-14T22:36:05.665ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
26132024-08-14T22:36:05.693ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
26142024-08-14T22:36:05.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 51527 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36576 req_id = 9fa69e7d-8142-4647-a590-983b4ab664f9 response_code = 204 uri = /switch-ports
26152024-08-14T22:36:05.717ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53040
26162024-08-14T22:36:05.718ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
26172024-08-14T22:36:05.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 787 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36576 req_id = aaa7705b-5f78-4d77-9d73-e51608ec440e response_code = 200 uri = /network-bootstore-config
26182024-08-14T22:36:05.744ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
26192024-08-14T22:36:05.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 50457 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53040 req_id = 9ee8707f-c506-4868-bb0a-b0c5a8ffdbc5 response_code = 204 uri = /switch-ports
26202024-08-14T22:36:05.768ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53040 req_id = 7b01c2e5-6756-4cfd-88dc-c2cc71276c2f response_code = 200 uri = /network-bootstore-config
26212024-08-14T22:36:24.306ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55566
26222024-08-14T22:36:24.306ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55566 req_id = f22e21dd-6a88-45cc-869b-27ab72ee1133 response_code = 200 uri = /baseboard
26232024-08-14T22:36:33.311ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 6812 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 79eb7b68-89d2-41ed-b3c7-79a8e36973ed response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
26242024-08-14T22:36:35.128ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45572
26252024-08-14T22:36:35.128ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45572 req_id = 41913af0-ed29-497c-bbfb-6cc083738e33 response_code = 200 uri = /v2p
26262024-08-14T22:36:35.194ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63768
26272024-08-14T22:36:35.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63768 req_id = 90591638-629a-4d8c-acc9-18d3ccfe6eb3 response_code = 200 uri = /vpc-routes
26282024-08-14T22:36:35.200ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63768 req_id = 3fcb81c3-c582-46a4-9bb4-8f195e83d45d response_code = 204 uri = /vpc-routes
26292024-08-14T22:36:35.279ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55646
26302024-08-14T22:36:35.280ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26312024-08-14T22:36:35.280ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26322024-08-14T22:36:35.280ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26332024-08-14T22:36:35.280ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26342024-08-14T22:36:35.280ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26352024-08-14T22:36:35.280ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26362024-08-14T22:36:35.280ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
26372024-08-14T22:36:35.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 952 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55646 req_id = 9e954ede-d23b-42fe-a4a7-87b2a8d96e12 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26382024-08-14T22:36:35.448ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59205
26392024-08-14T22:36:35.449ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
26402024-08-14T22:36:35.477ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
26412024-08-14T22:36:35.506ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 57893 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59205 req_id = fbc252f7-439e-4ead-94f0-c5a608c37517 response_code = 204 uri = /switch-ports
26422024-08-14T22:36:35.507ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57421
26432024-08-14T22:36:35.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1243 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57421 req_id = 2858ba78-ad36-4f82-ac0e-5f2d6047f4a4 response_code = 200 uri = /v2p
26442024-08-14T22:36:35.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59205 req_id = 6bbb4ae5-84ba-4f46-97d0-f740b2fbfc2e response_code = 200 uri = /network-bootstore-config
26452024-08-14T22:36:35.562ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33815
26462024-08-14T22:36:35.562ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33815 req_id = 746e9845-1bd9-4f78-891d-4b754e9c7bff response_code = 200 uri = /vpc-routes
26472024-08-14T22:36:35.569ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33815 req_id = f75128e0-b9fc-44af-9fe5-e26a6b94f014 response_code = 204 uri = /vpc-routes
26482024-08-14T22:36:35.700ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37343
26492024-08-14T22:36:35.700ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37343 req_id = ebe4326f-683c-4844-b9fb-9d959dbec987 response_code = 200 uri = /v2p
26502024-08-14T22:36:35.814ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52344
26512024-08-14T22:36:35.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 25 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52344 req_id = e04f5a22-62cb-4d70-8dc5-a9f6898431e5 response_code = 200 uri = /vpc-routes
26522024-08-14T22:36:35.822ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 818 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52344 req_id = 277b27d8-8cec-4581-8428-35d9f327df5f response_code = 204 uri = /vpc-routes
26532024-08-14T22:36:35.825ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55073
26542024-08-14T22:36:35.825ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26552024-08-14T22:36:35.825ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26562024-08-14T22:36:35.825ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26572024-08-14T22:36:35.825ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26582024-08-14T22:36:35.825ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26592024-08-14T22:36:35.825ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26602024-08-14T22:36:35.825ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
26612024-08-14T22:36:35.825ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 429 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55073 req_id = 08f02f6c-964a-4419-99ec-064d8099db9c response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26622024-08-14T22:36:35.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33754
26632024-08-14T22:36:35.844ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0327c40c-9b7a-419e-8572-976bbad0697f, kind: Service { id: 6f90e9b4-6395-4b60-a377-b8ab146c790e }, name: Name("external-dns-6f90e9b4-6395-4b60-a377-b8ab146c790e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 87d76b8e-70bc-4117-be4d-c0ec3b5b6c71, kind: Service { id: 2bd372ca-052a-4891-8e18-1e89afe09ee2 }, name: Name("external-dns-2bd372ca-052a-4891-8e18-1e89afe09ee2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 96])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 42428cc2-aa5d-4691-8f68-a5e7797bf030, kind: Service { id: de3c2053-cc1f-405f-977b-4cd7e79352bb }, name: Name("nexus-de3c2053-cc1f-405f-977b-4cd7e79352bb"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 92])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 685765c0-ff92-4e06-834e-6d6f9911b371, kind: Service { id: 30bf6548-3cd1-41c3-9086-faa8ca9474cf }, name: Name("nexus-30bf6548-3cd1-41c3-9086-faa8ca9474cf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 107])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b88f92ab-7657-4b1d-95d8-c07bcc49251f, kind: Service { id: b91e483c-c47c-4daa-a991-9e09554a7f98 }, name: Name("nexus-b91e483c-c47c-4daa-a991-9e09554a7f98"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26642024-08-14T22:36:35.844ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26652024-08-14T22:36:35.844ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26662024-08-14T22:36:35.844ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26672024-08-14T22:36:35.844ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26682024-08-14T22:36:35.844ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26692024-08-14T22:36:35.844ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
26702024-08-14T22:36:35.844ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1164 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33754 req_id = a0199ccf-a1ba-4dbd-9c66-498788315d1b response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26712024-08-14T22:36:35.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42181
26722024-08-14T22:36:35.897ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
26732024-08-14T22:36:35.924ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
26742024-08-14T22:36:35.947ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 50661 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:42181 req_id = c0f51b05-0e8a-41f1-a45c-a7ada509b8d7 response_code = 204 uri = /switch-ports
26752024-08-14T22:36:35.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 734 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42181 req_id = 709a52f5-fb76-4b28-9126-46ff8efa1203 response_code = 200 uri = /network-bootstore-config
26762024-08-14T22:36:35.973ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38594
26772024-08-14T22:36:35.973ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
26782024-08-14T22:36:35.999ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
26792024-08-14T22:36:36.022ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48912 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:38594 req_id = 90816c84-2f59-4647-a232-6a8cd5ce18cd response_code = 204 uri = /switch-ports
26802024-08-14T22:36:36.023ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38594 req_id = f0872cf3-5aae-4227-afde-e20e23e46df3 response_code = 200 uri = /network-bootstore-config
26812024-08-14T22:36:54.340ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33626
26822024-08-14T22:36:54.340ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33626 req_id = b81c231c-80fd-43b2-ab63-b11798edfbe7 response_code = 200 uri = /baseboard
26832024-08-14T22:37:03.311ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 6318 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = 4899fcc6-2b78-4adf-8523-6c06445e5e4f response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
26842024-08-14T22:37:04.903ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44372
26852024-08-14T22:37:04.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44372 req_id = 213cabaa-0ee6-40f2-9e64-7ae69cdc32ee response_code = 200 uri = /v2p
26862024-08-14T22:37:04.903ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37743
26872024-08-14T22:37:04.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37743 req_id = ca74c726-bf9d-4c8f-ac7a-ca8a92493f34 response_code = 200 uri = /vpc-routes
26882024-08-14T22:37:04.908ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37743 req_id = 1cdca8e6-91e9-4994-927e-0450d4130113 response_code = 204 uri = /vpc-routes
26892024-08-14T22:37:05.042ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56167
26902024-08-14T22:37:05.042ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
26912024-08-14T22:37:05.069ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
26922024-08-14T22:37:05.091ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49270 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56167 req_id = cafea098-bb1a-4f74-9c03-4291bed6ef03 response_code = 204 uri = /switch-ports
26932024-08-14T22:37:05.091ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56167 req_id = 599f7606-096c-4b3f-a952-53ddd30e3f44 response_code = 200 uri = /network-bootstore-config
26942024-08-14T22:37:05.466ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64026
26952024-08-14T22:37:05.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64026 req_id = df8872c6-3f2e-4d61-b7b1-0a2b01bb534e response_code = 200 uri = /v2p
26962024-08-14T22:37:05.471ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33917
26972024-08-14T22:37:05.471ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33917 req_id = 0f7cbb9d-9141-4a79-9a97-7c9d9b22f81d response_code = 200 uri = /vpc-routes
26982024-08-14T22:37:05.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33917 req_id = 53e891cb-ba29-412f-9c8c-e9a248457a5f response_code = 204 uri = /vpc-routes
26992024-08-14T22:37:05.546ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62411
27002024-08-14T22:37:05.546ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62411 req_id = 143a2fdc-c1dc-40aa-a167-33736bc79ba0 response_code = 200 uri = /v2p
27012024-08-14T22:37:05.624ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44767
27022024-08-14T22:37:05.624ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 20 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44767 req_id = 8439406c-8d81-4b62-bf71-d94e17a3f98b response_code = 200 uri = /vpc-routes
27032024-08-14T22:37:05.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44767 req_id = bd2c3267-f4d4-4267-a26e-4e024cc95064 response_code = 204 uri = /vpc-routes
27042024-08-14T22:37:05.679ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39958
27052024-08-14T22:37:05.679ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27062024-08-14T22:37:05.707ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27072024-08-14T22:37:05.729ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49755 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39958 req_id = 1eda3765-f0a6-4816-a946-30c8f6a9925b response_code = 204 uri = /switch-ports
27082024-08-14T22:37:05.729ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44069
27092024-08-14T22:37:05.729ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27102024-08-14T22:37:05.733ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 3379 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39958 req_id = 8c983eee-404c-4486-805b-f2d3bbf54bd9 response_code = 200 uri = /network-bootstore-config
27112024-08-14T22:37:05.753ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27122024-08-14T22:37:05.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 47525 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44069 req_id = 69deda0e-b2c3-489a-8d83-a0ef591f3ec2 response_code = 204 uri = /switch-ports
27132024-08-14T22:37:05.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44069 req_id = b0b55b79-e29d-496e-b5d0-430a474844fd response_code = 200 uri = /network-bootstore-config
27142024-08-14T22:37:24.373ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64711
27152024-08-14T22:37:24.373ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64711 req_id = 918e3aad-df0c-442f-91f9-b69c78f8aea3 response_code = 200 uri = /baseboard
27162024-08-14T22:37:33.314ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 8113 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = f118fce5-f44e-4931-9e62-115eb23863ca response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
27172024-08-14T22:37:33.671ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:303
27182024-08-14T22:37:33.671ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
27192024-08-14T22:37:33.705ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 12466, tv_nsec: 671023260 } time_to_next_cleanup = 599.99999557s
27202024-08-14T22:37:35.044ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53016
27212024-08-14T22:37:35.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53016 req_id = 3c350996-5401-4682-b2a0-3cf0e8c0389b response_code = 200 uri = /vpc-routes
27222024-08-14T22:37:35.045ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57787
27232024-08-14T22:37:35.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 623 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57787 req_id = 3776b0ca-e78d-4601-ae4e-27e5cd288ecd response_code = 200 uri = /v2p
27242024-08-14T22:37:35.052ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53016 req_id = edde8479-1563-4393-8323-c8a3fcb44ccd response_code = 204 uri = /vpc-routes
27252024-08-14T22:37:35.284ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33493
27262024-08-14T22:37:35.284ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27272024-08-14T22:37:35.315ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27282024-08-14T22:37:35.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 59462 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33493 req_id = 0c217678-fc7f-49a3-afd3-758068e14d12 response_code = 204 uri = /switch-ports
27292024-08-14T22:37:35.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33493 req_id = 365c4b28-e24b-453d-b5d9-6e8f39cd20e0 response_code = 200 uri = /network-bootstore-config
27302024-08-14T22:37:35.596ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47647
27312024-08-14T22:37:35.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47647 req_id = f4b35d6d-a31f-43a5-9370-727d2073f1ad response_code = 200 uri = /v2p
27322024-08-14T22:37:35.622ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39783
27332024-08-14T22:37:35.624ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39783 req_id = 793459ba-5f25-4f90-9070-33928488de1d response_code = 200 uri = /v2p
27342024-08-14T22:37:35.741ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60008
27352024-08-14T22:37:35.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 983 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60008 req_id = c261c8d2-58cc-4c4b-b53b-5a8573a5d79c response_code = 200 uri = /vpc-routes
27362024-08-14T22:37:35.748ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60008 req_id = 7575f02e-2ed9-449d-8005-b0b4d2c2e788 response_code = 204 uri = /vpc-routes
27372024-08-14T22:37:35.832ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56105
27382024-08-14T22:37:35.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56105 req_id = 464a0c48-db6c-4fb0-b63f-bfdb76c9045d response_code = 200 uri = /vpc-routes
27392024-08-14T22:37:35.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56105 req_id = 4300e7d1-b15a-4268-9bda-0ba2c3d16084 response_code = 204 uri = /vpc-routes
27402024-08-14T22:37:35.898ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43275
27412024-08-14T22:37:35.898ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27422024-08-14T22:37:35.926ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27432024-08-14T22:37:35.947ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 49732 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43275 req_id = 4602dc56-81db-47bf-9e17-24b16c12efab response_code = 204 uri = /switch-ports
27442024-08-14T22:37:35.950ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43275 req_id = 315d3690-070d-42a8-be48-fed9c80106b4 response_code = 200 uri = /network-bootstore-config
27452024-08-14T22:37:36.001ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59175
27462024-08-14T22:37:36.001ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27472024-08-14T22:37:36.026ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27482024-08-14T22:37:36.049ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 47839 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59175 req_id = 0fab56bf-d2aa-46a4-93b2-b7cd3add3c87 response_code = 204 uri = /switch-ports
27492024-08-14T22:37:36.049ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59175 req_id = c831684e-275f-4870-8131-8c4ebe31feec response_code = 200 uri = /network-bootstore-config
27502024-08-14T22:37:47.718ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
27512024-08-14T22:37:54.405ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49133
27522024-08-14T22:37:54.405ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 31 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49133 req_id = 98e56049-b095-4e95-982a-24a3e3dd13a1 response_code = 200 uri = /baseboard
27532024-08-14T22:38:03.398ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 7517 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = cb4f302a-2730-4010-af2d-309afa78aa53 response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
27542024-08-14T22:38:04.890ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43505
27552024-08-14T22:38:04.890ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43505 req_id = 762369bf-6557-40e3-bce9-f04b6623ba0a response_code = 200 uri = /vpc-routes
27562024-08-14T22:38:04.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43505 req_id = c767a4f9-5eb9-493b-ae98-bdad69470a18 response_code = 204 uri = /vpc-routes
27572024-08-14T22:38:04.920ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58061
27582024-08-14T22:38:04.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 631 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58061 req_id = 8e7e95a8-12e3-4043-82db-53be5341113b response_code = 200 uri = /v2p
27592024-08-14T22:38:05.041ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55499
27602024-08-14T22:38:05.041ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27612024-08-14T22:38:05.068ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27622024-08-14T22:38:05.093ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 52246 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:55499 req_id = ad5a6cc3-aa9f-4e49-8bc7-adef7fb3ae73 response_code = 204 uri = /switch-ports
27632024-08-14T22:38:05.093ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55499 req_id = 2ce4d27c-e8be-4c0d-845e-94cf4abaec3d response_code = 200 uri = /network-bootstore-config
27642024-08-14T22:38:05.494ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46962
27652024-08-14T22:38:05.494ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 217 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46962 req_id = 04e0835b-b0ee-43c9-88dc-009330bdec58 response_code = 200 uri = /v2p
27662024-08-14T22:38:05.571ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56389
27672024-08-14T22:38:05.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56389 req_id = 5cae5a65-e664-47b1-b57c-6605859b32bc response_code = 200 uri = /vpc-routes
27682024-08-14T22:38:05.578ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56389 req_id = b9f6f608-b070-4654-a920-9964042a1025 response_code = 204 uri = /vpc-routes
27692024-08-14T22:38:05.581ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49434
27702024-08-14T22:38:05.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49434 req_id = 71822b40-b2f2-4976-9c5a-b664c4ec91ab response_code = 200 uri = /v2p
27712024-08-14T22:38:05.627ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37176
27722024-08-14T22:38:05.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 867 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37176 req_id = dc144e83-fa1d-4216-aa2a-e0fc017c3f56 response_code = 200 uri = /vpc-routes
27732024-08-14T22:38:05.634ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37176 req_id = 91b716db-f468-4279-bf20-21f2c01f0b20 response_code = 204 uri = /vpc-routes
27742024-08-14T22:38:05.680ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48050
27752024-08-14T22:38:05.680ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27762024-08-14T22:38:05.708ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27772024-08-14T22:38:05.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 55788 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48050 req_id = b9901d0c-21e9-472d-b050-1be816ab57f9 response_code = 204 uri = /switch-ports
27782024-08-14T22:38:05.736ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49514
27792024-08-14T22:38:05.736ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
27802024-08-14T22:38:05.740ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 4421 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48050 req_id = 5b62eab2-4feb-49e6-976b-633d248a3673 response_code = 200 uri = /network-bootstore-config
27812024-08-14T22:38:05.762ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
27822024-08-14T22:38:05.784ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48106 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49514 req_id = 7fcc13bf-394c-4374-9940-4b46baaf647f response_code = 204 uri = /switch-ports
27832024-08-14T22:38:05.784ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49514 req_id = 43b794c3-9efa-47e4-a25f-5ce3df1e2a1b response_code = 200 uri = /network-bootstore-config
27842024-08-14T22:38:09.487ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40903
27852024-08-14T22:38:09.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 745 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40903 req_id = 22d3a130-ef51-4889-9ca3-083dd2803761 response_code = 200 uri = /v2p
27862024-08-14T22:38:09.488ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11357309) }
27872024-08-14T22:38:09.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40903 req_id = 80ef8603-30cd-4138-a62b-b375e9f43da2 response_code = 204 uri = /v2p
27882024-08-14T22:38:09.606ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51522
27892024-08-14T22:38:09.618ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:574 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(11357309))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-34b8ff8a493", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 52e8f999-25fb-4630-8978-3d9a5211af1e, read_only_parent: Some(Volume { block_size: 512, id: 2c0b29e2-66d8-4e87-ab5c-17f0fb7cbb48, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 997c5bc2-7cdd-4f92-9eac-675bd2f8d797, key: Some("VFFrDWB1J3JJo2wlCTm7WZ0/kfuv/SJmO1GXo8qrn0M="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::16]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 52e8f999-25fb-4630-8978-3d9a5211af1e, key: Some("GddQmFXmmCKpiOltl/dk7ppm5OWzdNVz4xkr289OqhY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19000", "[fd00:1122:3344:101::1a]:19003", "[fd00:1122:3344:101::17]:19002"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACcxLQOWQ5ZAADEtA5ZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAncS0DlkOWQAAxLQOWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJhZmYwYTFjOC0yNmE3LTRkZGUtODFiZC05NzFhZTk2ZjE2ZTQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTXFIY3V5d0dUaXRQMzhmaVlzYnBDNndReWJtQS9FNVBja3ZVSXc1MkliQiJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 instance_runtime = InstanceRuntimeState { propolis_id: Some(f092d045-558c-4423-881a-2caa10355712 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-14T22:38:09.302064918Z } metadata = InstanceMetadata { silo_id: 77456d4b-25be-4c3d-bb00-9759024c93b1, project_id: abf40c7e-7ed7-4517-9c18-330243d3c7e7 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = f092d045-558c-4423-881a-2caa10355712 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-14T22:38:09.270186Z }
27902024-08-14T22:38:09.618ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:608 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 (instance)
27912024-08-14T22:38:09.618ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1000 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 migration_id = None propolis_id = f092d045-558c-4423-881a-2caa10355712 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(11357309))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 51ffb8e6-9a35-417b-8928-8c7a8a3ba07f, kind: Instance { id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 188, 75])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11357309), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-34b8ff8a493", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 52e8f999-25fb-4630-8978-3d9a5211af1e, read_only_parent: Some(Volume { block_size: 512, id: 2c0b29e2-66d8-4e87-ab5c-17f0fb7cbb48, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 997c5bc2-7cdd-4f92-9eac-675bd2f8d797, key: Some("VFFrDWB1J3JJo2wlCTm7WZ0/kfuv/SJmO1GXo8qrn0M="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::16]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 52e8f999-25fb-4630-8978-3d9a5211af1e, key: Some("GddQmFXmmCKpiOltl/dk7ppm5OWzdNVz4xkr289OqhY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19000", "[fd00:1122:3344:101::1a]:19003", "[fd00:1122:3344:101::17]:19002"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACcxLQOWQ5ZAADEtA5ZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAncS0DlkOWQAAxLQOWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJhZmYwYTFjOC0yNmE3LTRkZGUtODFiZC05NzFhZTk2ZjE2ZTQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTXFIY3V5d0dUaXRQMzhmaVlzYnBDNndReWJtQS9FNVBja3ZVSXc1MkliQiJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-14T22:38:09.270186Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
27922024-08-14T22:38:09.619ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 12577 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51522 req_id = 05ef50c3-fab9-4cd2-a57f-ea64777235bb response_code = 200 uri = /instances/aff0a1c8-26a7-4dde-81bd-971ae96f16e4
27932024-08-14T22:38:09.727ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51039
27942024-08-14T22:38:09.728ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 846 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51039 req_id = dcc717e1-8a66-4925-bf58-d494dcdca5d4 response_code = 200 uri = /instances/aff0a1c8-26a7-4dde-81bd-971ae96f16e4/state
27952024-08-14T22:38:09.729ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 188, 75]), slot: 0, vni: Vni { inner: 11357309 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
27962024-08-14T22:38:09.729ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
27972024-08-14T22:38:09.767ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712 file = illumos-utils/src/zone.rs:316 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
27982024-08-14T22:38:09.790ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712 file = illumos-utils/src/zone.rs:354 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
27992024-08-14T22:38:10.991ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28002024-08-14T22:38:10.991ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712 zone file = illumos-utils/src/running_zone.rs:488 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28012024-08-14T22:38:11.176ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 40.692537ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28022024-08-14T22:38:11.228ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 131.023541ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28032024-08-14T22:38:11.380ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 199.710458ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28042024-08-14T22:38:11.598ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 418.876984ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28052024-08-14T22:38:12.033ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 821.775108ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28062024-08-14T22:38:12.866ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 558.668561ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28072024-08-14T22:38:13.437ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 1.460671771s file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28082024-08-14T22:38:14.909ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 1.172318096s file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 zone = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28092024-08-14T22:38:16.115ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712 file = sled-agent/src/instance.rs:1454 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28102024-08-14T22:38:16.125ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 54.364071ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28112024-08-14T22:38:16.189ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 77.319349ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28122024-08-14T22:38:16.275ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 247.728679ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28132024-08-14T22:38:16.532ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712") failed: Property not found. retry in 345.388052ms file = illumos-utils/src/svc.rs:36 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28142024-08-14T22:38:16.889ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1463 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28152024-08-14T22:38:16.926ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1487 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28162024-08-14T22:38:16.927ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACcxLQOWQ5ZAADEtA5ZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAncS0DlkOWQAAxLQOWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJhZmYwYTFjOC0yNmE3LTRkZGUtODFiZC05NzFhZTk2ZjE2ZTQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTXFIY3V5d0dUaXRQMzhmaVlzYnBDNndReWJtQS9FNVBja3ZVSXc1MkliQiJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-34b8ff8a493", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 52e8f999-25fb-4630-8978-3d9a5211af1e, read_only_parent: Some(Volume { block_size: 512, id: 2c0b29e2-66d8-4e87-ab5c-17f0fb7cbb48, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 997c5bc2-7cdd-4f92-9eac-675bd2f8d797, key: Some("VFFrDWB1J3JJo2wlCTm7WZ0/kfuv/SJmO1GXo8qrn0M="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::16]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 52e8f999-25fb-4630-8978-3d9a5211af1e, key: Some("GddQmFXmmCKpiOltl/dk7ppm5OWzdNVz4xkr289OqhY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19000", "[fd00:1122:3344:101::1a]:19003", "[fd00:1122:3344:101::17]:19002"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "opte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: aff0a1c8-26a7-4dde-81bd-971ae96f16e4, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: abf40c7e-7ed7-4517-9c18-330243d3c7e7, silo_id: 77456d4b-25be-4c3d-bb00-9759024c93b1, sled_id: aeb7e077-0751-417b-8b16-b62876b5611c, sled_model: "i86pc", sled_revision: 0, sled_serial: "buskin" }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:736 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28172024-08-14T22:38:17.064ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:738 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28182024-08-14T22:38:17.068ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-08-14T22:38:17.068390384Z } file = sled-agent/src/instance.rs:606 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28192024-08-14T22:38:17.068ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:628 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-14T22:38:17.068390384Z } propolis_id = f092d045-558c-4423-881a-2caa10355712
28202024-08-14T22:38:17.068ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:525 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 state = SledInstanceState { propolis_id: f092d045-558c-4423-881a-2caa10355712 (propolis), vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-14T22:38:17.068390384Z }, migration_in: None, migration_out: None }
28212024-08-14T22:38:17.107ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-08-14T22:38:17.107611017Z } file = sled-agent/src/instance.rs:606 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28222024-08-14T22:38:17.107ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:628 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-14T22:38:17.107611017Z } propolis_id = f092d045-558c-4423-881a-2caa10355712
28232024-08-14T22:38:17.107ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:525 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 state = SledInstanceState { propolis_id: f092d045-558c-4423-881a-2caa10355712 (propolis), vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-14T22:38:17.107611017Z }, migration_in: None, migration_out: None }
28242024-08-14T22:38:24.438ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39994
28252024-08-14T22:38:24.439ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 625 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39994 req_id = 8b105e1c-8956-40df-ab5b-f3a87be675c5 response_code = 200 uri = /baseboard
28262024-08-14T22:38:33.398ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 7233 local_addr = [fd00:1122:3344:101::1]:49293 method = GET remote_addr = [fd00:1122:3344:101::d]:48533 req_id = a507f49d-ee4b-46b9-bb8d-aaf9b08d8514 response_code = 200 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c uri = /aeb7e077-0751-417b-8b16-b62876b5611c
28272024-08-14T22:38:35.008ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48708
28282024-08-14T22:38:35.008ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48708 req_id = a8d10d21-b92e-4b43-9be3-62f92bdbe0d9 response_code = 200 uri = /instances/aff0a1c8-26a7-4dde-81bd-971ae96f16e4/state
28292024-08-14T22:38:35.127ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65495
28302024-08-14T22:38:35.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65495 req_id = 36846fa3-cfa1-4985-80c5-11251cebe9cd response_code = 200 uri = /vpc-routes
28312024-08-14T22:38:35.150ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63312
28322024-08-14T22:38:35.150ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63312 req_id = 8b38acf8-9bef-44f1-b005-088904c42737 response_code = 200 uri = /v2p
28332024-08-14T22:38:35.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 315 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:65495 req_id = bdb8d6fa-21da-419c-92bc-105709a6bd71 response_code = 204 uri = /vpc-routes
28342024-08-14T22:38:35.391ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65134
28352024-08-14T22:38:35.391ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
28362024-08-14T22:38:35.422ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
28372024-08-14T22:38:35.452ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 61415 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:65134 req_id = e0f96ff7-618d-41c2-b888-0a7bcc141727 response_code = 204 uri = /switch-ports
28382024-08-14T22:38:35.454ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1286 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65134 req_id = 3e645956-353c-42dc-9968-5e62f0dbaa25 response_code = 200 uri = /network-bootstore-config
28392024-08-14T22:38:35.617ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37433
28402024-08-14T22:38:35.617ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37433 req_id = c4dc62f9-0695-44fd-9c2b-671ca1a477c0 response_code = 200 uri = /v2p
28412024-08-14T22:38:35.759ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57241
28422024-08-14T22:38:35.760ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1055 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57241 req_id = 6ef3c750-fab8-4f8c-8d39-368d2ce43b34 response_code = 200 uri = /instances/aff0a1c8-26a7-4dde-81bd-971ae96f16e4/state
28432024-08-14T22:38:35.771ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47446
28442024-08-14T22:38:35.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47446 req_id = c2f63061-235d-44e4-8464-f75552230b1a response_code = 200 uri = /vpc-routes
28452024-08-14T22:38:35.786ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 1081 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47446 req_id = 91edce76-ec9f-4b2c-8dbd-17dd9db0cfbb response_code = 204 uri = /vpc-routes
28462024-08-14T22:38:35.813ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53102
28472024-08-14T22:38:35.813ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53102 req_id = 89fdc228-b7b8-4867-8c51-219a0367db71 response_code = 200 uri = /instances/aff0a1c8-26a7-4dde-81bd-971ae96f16e4/state
28482024-08-14T22:38:35.819ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47337
28492024-08-14T22:38:35.819ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47337 req_id = 27c86ae9-0de2-4a03-b9f1-859ecf0aee80 response_code = 200 uri = /v2p
28502024-08-14T22:38:35.863ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34938
28512024-08-14T22:38:35.863ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 31 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34938 req_id = a66788c9-b6cc-47da-aad3-3fcf453ffee5 response_code = 200 uri = /vpc-routes
28522024-08-14T22:38:35.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34938 req_id = 2b53b21c-78cc-41dd-a8c3-33798d3115be response_code = 204 uri = /vpc-routes
28532024-08-14T22:38:35.928ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65328
28542024-08-14T22:38:35.928ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
28552024-08-14T22:38:35.955ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
28562024-08-14T22:38:35.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 52363 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65328 req_id = 44b2455d-d6ec-4f8c-b4a7-95a9c6f4d5a9 response_code = 204 uri = /switch-ports
28572024-08-14T22:38:35.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65328 req_id = 1e21e07b-fb93-4e9d-97ec-adb769277a1b response_code = 200 uri = /network-bootstore-config
28582024-08-14T22:38:36.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36200
28592024-08-14T22:38:36.148ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3937
28602024-08-14T22:38:36.174ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3948
28612024-08-14T22:38:36.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 48714 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:36200 req_id = f89ad3d2-baa9-434e-ab23-e473a31afe05 response_code = 204 uri = /switch-ports
28622024-08-14T22:38:36.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36200 req_id = 400f9e8c-dee9-4cdb-a459-acfa96c5b236 response_code = 200 uri = /network-bootstore-config
28632024-08-14T22:38:43.155ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40986
28642024-08-14T22:38:43.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 566 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40986 req_id = 5d012c2a-6b20-44ad-89e6-951d0aa4e44d response_code = 200 uri = /instances/aff0a1c8-26a7-4dde-81bd-971ae96f16e4/state
28652024-08-14T22:38:43.156ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-08-14T22:38:43.156704349Z } file = sled-agent/src/instance.rs:606 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28662024-08-14T22:38:43.156ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:628 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-14T22:38:43.156704349Z } propolis_id = f092d045-558c-4423-881a-2caa10355712
28672024-08-14T22:38:43.156ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:525 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 state = SledInstanceState { propolis_id: f092d045-558c-4423-881a-2caa10355712 (propolis), vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-14T22:38:43.156704349Z }, migration_in: None, migration_out: None }
28682024-08-14T22:38:43.174ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-08-14T22:38:43.174560859Z } file = sled-agent/src/instance.rs:606 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28692024-08-14T22:38:43.174ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:628 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-08-14T22:38:43.174560859Z } propolis_id = f092d045-558c-4423-881a-2caa10355712
28702024-08-14T22:38:43.174ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:642 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28712024-08-14T22:38:43.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28722024-08-14T22:38:43.174ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 12055948051223197284 n_samples = 6 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
28732024-08-14T22:38:43.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28742024-08-14T22:38:43.174ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/2e8af2e9-8342-42ed-88bb-bcb6bcd354d3/debug/bundle/zone", "/pool/int/eb7d1608-061f-4633-a5da-2133d1a63302/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/17668df1-993b-458e-b1cf-310a88477d11/crypt/debug", "/pool/ext/7f6d9db3-762e-4798-98cd-bb00d0a3b252/crypt/debug", "/pool/ext/1bd3ff04-304c-4362-b41f-8347b5dd9341/crypt/debug", "/pool/ext/db606de8-ae75-4597-bde9-3f1234d88ff3/crypt/debug", "/pool/ext/20226c3a-235f-4118-b164-e14967848f5c/crypt/debug", "/pool/ext/2450844a-e7c2-49d0-8dc6-d59793fa3c37/crypt/debug", "/pool/ext/a4af3495-6c0f-4fe3-b8a0-503d2a880027/crypt/debug", "/pool/ext/0faf5893-5a63-4906-a7f5-cb36c6c1353b/crypt/debug", "/pool/ext/126123c3-ef9f-486e-8f31-7272296cc4d8/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:445 zone_name = oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712
28752024-08-14T22:38:43.174ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 1981189877533013973 n_samples = 6 sled_id = aeb7e077-0751-417b-8b16-b62876b5611c
28762024-08-14T22:38:43.801ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1249 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712", bundle_id: 3c1cf89c-5054-417a-99df-dddb85da328c }, time_created: 2024-08-14T22:38:43.175255490Z, version: 0, cause: TerminatedInstance }
28772024-08-14T22:38:43.801ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_f092d045-558c-4423-881a-2caa10355712 file = sled-agent/src/instance.rs:837 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28782024-08-14T22:38:45.547ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28792024-08-14T22:38:45.563ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(11357309), kind: System }
28802024-08-14T22:38:45.563ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(11357309), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
28812024-08-14T22:38:45.563ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:525 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 state = SledInstanceState { propolis_id: f092d045-558c-4423-881a-2caa10355712 (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-14T22:38:45.563653672Z }, migration_in: None, migration_out: None }
28822024-08-14T22:38:45.584ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:525 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4 state = SledInstanceState { propolis_id: f092d045-558c-4423-881a-2caa10355712 (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-14T22:38:45.563653672Z }, migration_in: None, migration_out: None }
28832024-08-14T22:38:45.588ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:774 instance_id = aff0a1c8-26a7-4dde-81bd-971ae96f16e4
28842024-08-14T22:38:45.963ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57054
28852024-08-14T22:38:45.963ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 521 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57054 req_id = 6859a74d-8005-4a12-821e-f747770261e0 response_code = 200 uri = /vpc-routes
28862024-08-14T22:38:45.969ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57054 req_id = 1bb6080f-cf18-4de3-920a-c71dd5139599 response_code = 204 uri = /vpc-routes
28872024-08-14T22:38:45.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48123
28882024-08-14T22:38:45.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48123 req_id = 6d6afd06-3edb-44df-a109-4a8be50704af response_code = 200 uri = /v2p
28892024-08-14T22:38:46.077ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54652
28902024-08-14T22:38:46.077ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54652 req_id = 484fa05e-a0ad-46d9-ada3-221940fb4b76 response_code = 200 uri = /v2p
28912024-08-14T22:38:47.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38247
28922024-08-14T22:38:47.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38247 req_id = afb60d66-a978-48d9-b14d-c50150fb7e09 response_code = 200 uri = /vpc-routes
28932024-08-14T22:38:47.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/52d900a/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38247 req_id = 6d82ef6b-06eb-4b77-9add-4e24ab2012ca response_code = 204 uri = /vpc-routes