Max level shown:
1 [ Oct 18 17:54:07 Enabled. ]
2 [ Oct 18 17:54:07 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Oct 18 17:54:07 Method "start" exited with status 0. ]
4 [ Oct 18 17:54:07 Rereading configuration. ]
5 [ Oct 18 17:54:07 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-10-18T17:54:07.132ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-10-18T17:54:07.132ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-10-18T17:54:07.185ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-10-18T17:54:07.185ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-10-18T17:54:07.296ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-10-18T17:54:07.305ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-10-18T17:54:07.312ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-10-18T17:54:07.914ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-10-18T17:54:08.051ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-10-18T17:54:08.051ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-10-18T17:54:08.051ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-10-18T17:54:08.051ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-10-18T17:54:08.051ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-10-18T17:54:08.051ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-10-18T17:54:08.054ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-10-18T17:54:08.054ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-10-18T17:54:08.054ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-10-18T17:54:08.054ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-10-18T17:54:08.054ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-10-18T17:54:08.054ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-10-18T17:54:08.054ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-10-18T17:54:08.054ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-10-18T17:54:08.054ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-10-18T17:54:08.064ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-10-18T17:54:08.064ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool)
322024-10-18T17:54:08.160ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal }
332024-10-18T17:54:08.538ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-10-18T17:54:08.538ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-10-18T17:54:08.538ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-10-18T17:54:08.538ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-10-18T17:54:08.538ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-10-18T17:54:08.538ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-10-18T17:54:08.538ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-10-18T17:54:08.547ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412024-10-18T17:54:08.547ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 62298c00-fb90-4e92-a4b4-95bca961978c (zpool)
422024-10-18T17:54:08.629ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal }
432024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
532024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
542024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
602024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
632024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
642024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
652024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
662024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
702024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
712024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
722024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
732024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
782024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
822024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-10-18T17:54:09.027ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
882024-10-18T17:54:09.027ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
892024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-10-18T17:54:09.027ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-10-18T17:54:09.027ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-10-18T17:54:09.027ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-10-18T17:54:09.028ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-10-18T17:54:09.028ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-10-18T17:54:09.028ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-10-18T17:54:09.028ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-10-18T17:54:09.028ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-10-18T17:54:09.028ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-10-18T17:54:09.028ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1002024-10-18T17:54:09.028ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1012024-10-18T17:54:09.028ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1022024-10-18T17:54:09.028ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-10-18T17:54:09.028ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-10-18T17:54:09.028ZINFOSledAgent (bootstore): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-10-18T17:54:09.028ZINFOSledAgent (bootstore): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-10-18T17:54:09.028ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-10-18T17:54:09.028ZINFOSledAgent (bootstore): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-10-18T17:54:09.028ZINFOSledAgent (bootstore): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-10-18T17:54:09.028ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-10-18T17:54:09.028ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-10-18T17:54:09.028ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-10-18T17:54:09.038ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:739
1132024-10-18T17:54:09.038ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1786, tv_nsec: 706811491 } time_to_next_cleanup = 599.999923629s
1142024-10-18T17:54:09.038ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3924
1152024-10-18T17:54:09.038ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-10-18T17:54:09.039ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-10-18T17:54:09.041ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-10-18T17:54:09.041ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-10-18T17:54:09.043ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-10-18T17:54:09.044ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1073
1212024-10-18T17:54:09.044ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1222024-10-18T17:54:09.044ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-10-18T17:54:09.044ZINFOSledAgent (RSS): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-10-18T17:54:09.044ZINFOSledAgent (RSS): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-10-18T17:54:09.044ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1136
1262024-10-18T17:54:09.044ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-10-18T17:54:09.044ZINFOSledAgent (RSS): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-10-18T17:54:09.044ZINFOSledAgent (RSS): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-10-18T17:54:09.044ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1178
1302024-10-18T17:54:09.044ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-10-18T17:54:09.044ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-10-18T17:54:09.044ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-10-18T17:54:09.044ZINFOSledAgent (RSS): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-10-18T17:54:09.044ZINFOSledAgent (RSS): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-10-18T17:54:09.044ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 33bc13e3-6beb-4d1b-8b04-6e3da281e5d7, sleds: { [fdb0:18c0:4d0d:9fb2::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 08066c3d-b7ed-49d2-8578-f8769fb94736 (sled), rack_id: 33bc13e3-6beb-4d1b-8b04-6e3da281e5d7, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-10-18T17:54:09.044ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1212
1372024-10-18T17:54:09.045ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-10-18T17:54:09.045ZINFOSledAgent (bootstore): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-10-18T17:54:09.045ZINFOSledAgent (bootstore): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-10-18T17:54:09.045ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-10-18T17:54:09.045ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 08066c3d-b7ed-49d2-8578-f8769fb94736 (sled), rack_id: 33bc13e3-6beb-4d1b-8b04-6e3da281e5d7, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-10-18T17:54:09.046ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4221
1432024-10-18T17:54:09.046ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4603
1442024-10-18T17:54:09.050ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-10-18T17:54:09.056ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-10-18T17:54:09.056ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-10-18T17:54:09.060ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-10-18T17:54:09.060ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-10-18T17:54:09.060ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:43269
1502024-10-18T17:54:09.060ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 08066c3d-b7ed-49d2-8578-f8769fb94736 (sled), rack_id: 33bc13e3-6beb-4d1b-8b04-6e3da281e5d7, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1512024-10-18T17:54:09.060ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1522024-10-18T17:54:09.060ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1532024-10-18T17:54:09.060ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1542024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1552024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1562024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1572024-10-18T17:54:09.060ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1582024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-10-18T17:54:09.060ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-10-18T17:54:09.060ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-10-18T17:54:09.076ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-10-18T17:54:09.097ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-10-18T17:54:09.114ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1672024-10-18T17:54:09.136ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
1682024-10-18T17:54:09.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1692024-10-18T17:54:09.136ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
1702024-10-18T17:54:09.136ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1712024-10-18T17:54:09.140ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 134.159979ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0fa141d6-4659-4da9-a1aa-cf3a1c4b5928", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0fa141d6-4659-4da9-a1aa-cf3a1c4b5928" } file = clients/ddm-admin-client/src/lib.rs:177
1722024-10-18T17:54:09.150ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_e40afb95-7af8-4f62-aab4-e63b3bd02855/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-10-18T17:54:09.175ZINFOSledAgent: successfully created encrypted zvol: name="oxi_e40afb95-7af8-4f62-aab4-e63b3bd02855/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-10-18T17:54:09.175ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_e40afb95-7af8-4f62-aab4-e63b3bd02855/swap" file = sled-agent/src/swap_device.rs:96
1752024-10-18T17:54:09.180ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
1762024-10-18T17:54:09.180ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1772024-10-18T17:54:09.180ZINFOSledAgent (BackingFs): Ensuring dataset oxi_e40afb95-7af8-4f62-aab4-e63b3bd02855/backing/fmd file = sled-agent/src/backing_fs.rs:138
1782024-10-18T17:54:09.266ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1792024-10-18T17:54:09.275ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1802024-10-18T17:54:09.275ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 68.021879ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3e614c31-b51e-4af6-b762-8b1ccc0b1e0f", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3e614c31-b51e-4af6-b762-8b1ccc0b1e0f" } file = clients/ddm-admin-client/src/lib.rs:177
1812024-10-18T17:54:09.344ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-10-18T17:54:09.345ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 85.559323ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7c9ad2eb-f720-4c13-820b-b1915f37c927", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7c9ad2eb-f720-4c13-820b-b1915f37c927" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-10-18T17:54:09.431ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-10-18T17:54:09.431ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 211.997774ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "df9f8ff3-38a4-4f61-a7cc-e7cf72e315e0", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "df9f8ff3-38a4-4f61-a7cc-e7cf72e315e0" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-10-18T17:54:09.485ZINFOSledAgent (BackingFs): Mounting oxi_e40afb95-7af8-4f62-aab4-e63b3bd02855/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1862024-10-18T17:54:09.498ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1872024-10-18T17:54:09.498ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1882024-10-18T17:54:09.498ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1892024-10-18T17:54:09.499ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1902024-10-18T17:54:09.646ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-10-18T17:54:09.646ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 148.571206ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d4933168-46cb-454b-9933-542f70f052f6", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d4933168-46cb-454b-9933-542f70f052f6" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-10-18T17:54:09.796ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-10-18T17:54:09.797ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 308.691644ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8f310fee-31df-4d00-8b59-80424d1d6659", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8f310fee-31df-4d00-8b59-80424d1d6659" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-10-18T17:54:10.107ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-10-18T17:54:10.108ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 405.034983ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "450434da-4d73-4bcc-8ca9-c95b060ffdee", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "450434da-4d73-4bcc-8ca9-c95b060ffdee" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-10-18T17:54:10.220ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
1972024-10-18T17:54:10.225ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:65402 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
1982024-10-18T17:54:10.226ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
1992024-10-18T17:54:10.226ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:65402 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2002024-10-18T17:54:10.227ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 211.41666ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2012024-10-18T17:54:10.227ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2022024-10-18T17:54:10.227ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2032024-10-18T17:54:10.228ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:923 underlay_address = fd00:1122:3344:101::1
2042024-10-18T17:54:10.228ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
2052024-10-18T17:54:10.439ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 561.000161ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2062024-10-18T17:54:10.514ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2072024-10-18T17:54:11.002ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 924.614722ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2082024-10-18T17:54:11.927ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.192787676s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2092024-10-18T17:54:12.641ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2679
2102024-10-18T17:54:12.641ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2897
2112024-10-18T17:54:12.641ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2549
2122024-10-18T17:54:12.641ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2596
2132024-10-18T17:54:12.641ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2978
2142024-10-18T17:54:12.641ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3020
2152024-10-18T17:54:12.643ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2822
2162024-10-18T17:54:12.644ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2590
2172024-10-18T17:54:12.644ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='e2c94a14-1106-4e76-8538-bace2c8fb163'/> <propval type="astring" name="rack_id" value='33bc13e3-6beb-4d1b-8b04-6e3da281e5d7'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe07:3200'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='08066c3d-b7ed-49d2-8578-f8769fb94736'/> <propval type="astring" name="rack_id" value='33bc13e3-6beb-4d1b-8b04-6e3da281e5d7'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='08066c3d-b7ed-49d2-8578-f8769fb94736'/> <propval type="astring" name="rack_uuid" value='33bc13e3-6beb-4d1b-8b04-6e3da281e5d7'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='08066c3d-b7ed-49d2-8578-f8769fb94736'/> <propval type="astring" name="rack_uuid" value='33bc13e3-6beb-4d1b-8b04-6e3da281e5d7'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2182024-10-18T17:54:12.645ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2192024-10-18T17:54:12.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 34.173026ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-10-18T17:54:13.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 50.871401ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-10-18T17:54:13.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 150.07291ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-10-18T17:54:13.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 480.004915ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-10-18T17:54:13.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 759.173194ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-10-18T17:54:14.122ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.556843169s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2252024-10-18T17:54:14.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 581.431101ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-10-18T17:54:15.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.274523035s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-10-18T17:54:16.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.292749205s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2282024-10-18T17:54:18.563ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:12345
2292024-10-18T17:54:18.564ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2302024-10-18T17:54:18.564ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2312024-10-18T17:54:18.564ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2322024-10-18T17:54:18.564ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2332024-10-18T17:54:18.564ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2342024-10-18T17:54:18.564ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2352024-10-18T17:54:18.564ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3924
2362024-10-18T17:54:18.564ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:43269
2372024-10-18T17:54:18.807ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2382024-10-18T17:54:18.807ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2392024-10-18T17:54:18.807ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2402024-10-18T17:54:18.807ZINFOSledAgent (RSS): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2412024-10-18T17:54:18.807ZINFOSledAgent (RSS): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2422024-10-18T17:54:18.807ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2432024-10-18T17:54:18.807ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-10-18T17:54:18.807ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-10-18T17:54:18.809ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 5.193411734s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2462024-10-18T17:54:18.813ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4245 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2472024-10-18T17:54:18.813ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4261
2482024-10-18T17:54:18.813ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2492024-10-18T17:54:19.075ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38982
2502024-10-18T17:54:19.075ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2512024-10-18T17:54:19.075ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2522024-10-18T17:54:19.075ZINFOSledAgent (ServiceManager): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:188
2532024-10-18T17:54:19.075ZINFOSledAgent (ServiceManager): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:188
2542024-10-18T17:54:19.075ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 387 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38982 req_id = 0e150506-f281-42fb-89a4-512ec3728017 response_code = 200 uri = /inventory
2552024-10-18T17:54:19.106ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45738
2562024-10-18T17:54:19.107ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 30 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45738 req_id = 0a3c46f8-483d-4327-a188-76b6a05b3799 response_code = 200 uri = /sled-role
2572024-10-18T17:54:19.108ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2582024-10-18T17:54:19.108ZINFOSledAgent (RSS): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2592024-10-18T17:54:19.108ZINFOSledAgent (RSS): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2602024-10-18T17:54:19.109ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:882
2612024-10-18T17:54:19.138ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 20335afa-6eea-4f23-a3f5-8df78c6c8e50, pool_id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: c26a1128-eca4-432a-a9e7-cab8cc50a3cb, pool_id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: ee2d4b24-e1e6-4689-8de5-2a74c881666f, pool_id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 1838a00e-8d64-49d4-8b47-216dfa7830a8, pool_id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 37b21758-d6b3-4a32-946c-b6c04b417dd7, pool_id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: f6b6c54a-2880-47d3-9451-12d39430d203, pool_id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 5d4a1bc6-9204-44c2-8b9e-80a1dd662d32, pool_id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 74d5a544-0c0e-49bf-8cff-fb634583f926, pool_id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 4e7cff97-1fa7-4eba-8866-b612f9be3bc3, pool_id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2622024-10-18T17:54:19.138ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60647
2632024-10-18T17:54:19.138ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:864 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2642024-10-18T17:54:19.138ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 20335afa-6eea-4f23-a3f5-8df78c6c8e50, pool_id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: c26a1128-eca4-432a-a9e7-cab8cc50a3cb, pool_id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: ee2d4b24-e1e6-4689-8de5-2a74c881666f, pool_id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 1838a00e-8d64-49d4-8b47-216dfa7830a8, pool_id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 37b21758-d6b3-4a32-946c-b6c04b417dd7, pool_id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: f6b6c54a-2880-47d3-9451-12d39430d203, pool_id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 5d4a1bc6-9204-44c2-8b9e-80a1dd662d32, pool_id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 74d5a544-0c0e-49bf-8cff-fb634583f926, pool_id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 4e7cff97-1fa7-4eba-8866-b612f9be3bc3, pool_id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2652024-10-18T17:54:19.138ZINFOSledAgent (StorageManager): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2662024-10-18T17:54:19.138ZINFOSledAgent (StorageManager): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2672024-10-18T17:54:19.138ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2682024-10-18T17:54:19.138ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2692024-10-18T17:54:19.138ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2702024-10-18T17:54:19.138ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2712024-10-18T17:54:19.138ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2722024-10-18T17:54:19.295ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2732024-10-18T17:54:19.296ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 8853a1c1-d102-4f80-9012-356473e594ff (zpool)
2742024-10-18T17:54:19.566ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/08066c3d-b7ed-49d2-8578-f8769fb94736) file = sled-agent/src/probe_manager.rs:230 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2752024-10-18T17:54:20.261ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 291083675488840079 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2762024-10-18T17:54:20.261ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7620151230303063215 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2772024-10-18T17:54:20.488ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2782024-10-18T17:54:20.500ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }
2792024-10-18T17:54:20.508ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2802024-10-18T17:54:20.508ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2812024-10-18T17:54:20.508ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2822024-10-18T17:54:20.508ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2832024-10-18T17:54:20.508ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2842024-10-18T17:54:20.508ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2852024-10-18T17:54:20.508ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2862024-10-18T17:54:20.542ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2872024-10-18T17:54:20.549ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8853a1c1-d102-4f80-9012-356473e594ff/crypt/zone file = sled-storage/src/dataset.rs:280
2882024-10-18T17:54:20.570ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/08066c3d-b7ed-49d2-8578-f8769fb94736) file = sled-agent/src/probe_manager.rs:230 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
2892024-10-18T17:54:20.716ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8853a1c1-d102-4f80-9012-356473e594ff file = sled-storage/src/dataset.rs:366
2902024-10-18T17:54:20.725ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2912024-10-18T17:54:20.725ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2922024-10-18T17:54:20.725ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2932024-10-18T17:54:20.725ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2942024-10-18T17:54:20.725ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2952024-10-18T17:54:20.734ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2962024-10-18T17:54:20.734ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool)
2972024-10-18T17:54:20.827ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }
2982024-10-18T17:54:20.835ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2992024-10-18T17:54:20.835ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
3002024-10-18T17:54:20.835ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
3012024-10-18T17:54:20.835ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
3022024-10-18T17:54:20.835ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3032024-10-18T17:54:20.835ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3042024-10-18T17:54:20.835ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3052024-10-18T17:54:20.870ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3062024-10-18T17:54:20.878ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8523f2c5-b045-4f20-8f22-89f361c6e33c/crypt/zone file = sled-storage/src/dataset.rs:280
3072024-10-18T17:54:21.056ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8523f2c5-b045-4f20-8f22-89f361c6e33c file = sled-storage/src/dataset.rs:366
3082024-10-18T17:54:21.064ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3092024-10-18T17:54:21.064ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3102024-10-18T17:54:21.064ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3112024-10-18T17:54:21.064ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3122024-10-18T17:54:21.064ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3132024-10-18T17:54:21.073ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3142024-10-18T17:54:21.073ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 24197587-83af-46c0-a36e-5e4a26350d21 (zpool)
3152024-10-18T17:54:21.162ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }
3162024-10-18T17:54:21.168ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3172024-10-18T17:54:21.168ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3182024-10-18T17:54:21.168ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3192024-10-18T17:54:21.168ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3202024-10-18T17:54:21.168ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3212024-10-18T17:54:21.168ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3222024-10-18T17:54:21.169ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3232024-10-18T17:54:21.203ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3242024-10-18T17:54:21.211ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_24197587-83af-46c0-a36e-5e4a26350d21/crypt/zone file = sled-storage/src/dataset.rs:280
3252024-10-18T17:54:21.380ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_24197587-83af-46c0-a36e-5e4a26350d21 file = sled-storage/src/dataset.rs:366
3262024-10-18T17:54:21.388ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3272024-10-18T17:54:21.388ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3282024-10-18T17:54:21.388ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3292024-10-18T17:54:21.388ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3302024-10-18T17:54:21.389ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3312024-10-18T17:54:21.398ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3322024-10-18T17:54:21.398ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool)
3332024-10-18T17:54:21.493ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }
3342024-10-18T17:54:21.501ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3352024-10-18T17:54:21.501ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3362024-10-18T17:54:21.501ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3372024-10-18T17:54:21.501ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3382024-10-18T17:54:21.501ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3392024-10-18T17:54:21.501ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3402024-10-18T17:54:21.501ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3412024-10-18T17:54:21.536ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3422024-10-18T17:54:21.544ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_956d8649-5e02-4c83-a766-709d6c1c19f4/crypt/zone file = sled-storage/src/dataset.rs:280
3432024-10-18T17:54:21.572ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/08066c3d-b7ed-49d2-8578-f8769fb94736) file = sled-agent/src/probe_manager.rs:230 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
3442024-10-18T17:54:21.708ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_956d8649-5e02-4c83-a766-709d6c1c19f4 file = sled-storage/src/dataset.rs:366
3452024-10-18T17:54:21.715ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3462024-10-18T17:54:21.715ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3472024-10-18T17:54:21.715ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3482024-10-18T17:54:21.715ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3492024-10-18T17:54:21.715ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3502024-10-18T17:54:21.724ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3512024-10-18T17:54:21.724ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool)
3522024-10-18T17:54:21.816ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }
3532024-10-18T17:54:21.823ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3542024-10-18T17:54:21.824ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3552024-10-18T17:54:21.824ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3562024-10-18T17:54:21.824ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3572024-10-18T17:54:21.824ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3582024-10-18T17:54:21.824ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3592024-10-18T17:54:21.824ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3602024-10-18T17:54:21.860ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3612024-10-18T17:54:21.869ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4e3d3772-4d1a-4cd4-9451-d3f5402e0376/crypt/zone file = sled-storage/src/dataset.rs:280
3622024-10-18T17:54:22.043ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4e3d3772-4d1a-4cd4-9451-d3f5402e0376 file = sled-storage/src/dataset.rs:366
3632024-10-18T17:54:22.051ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3642024-10-18T17:54:22.051ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3652024-10-18T17:54:22.051ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3662024-10-18T17:54:22.051ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3672024-10-18T17:54:22.051ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3682024-10-18T17:54:22.060ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3692024-10-18T17:54:22.060ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = aa3e117f-01a2-4173-8076-71ed9651de49 (zpool)
3702024-10-18T17:54:22.152ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }
3712024-10-18T17:54:22.160ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3722024-10-18T17:54:22.160ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3732024-10-18T17:54:22.160ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3742024-10-18T17:54:22.160ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3752024-10-18T17:54:22.160ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3762024-10-18T17:54:22.160ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3772024-10-18T17:54:22.160ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3782024-10-18T17:54:22.193ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3792024-10-18T17:54:22.200ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_aa3e117f-01a2-4173-8076-71ed9651de49/crypt/zone file = sled-storage/src/dataset.rs:280
3802024-10-18T17:54:22.376ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_aa3e117f-01a2-4173-8076-71ed9651de49 file = sled-storage/src/dataset.rs:366
3812024-10-18T17:54:22.384ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3822024-10-18T17:54:22.385ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3832024-10-18T17:54:22.385ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3842024-10-18T17:54:22.385ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3852024-10-18T17:54:22.385ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3862024-10-18T17:54:22.394ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3872024-10-18T17:54:22.394ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = e09cb849-5302-41dd-87a5-36e8723c48fb (zpool)
3882024-10-18T17:54:22.493ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }
3892024-10-18T17:54:22.501ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3902024-10-18T17:54:22.501ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3912024-10-18T17:54:22.501ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3922024-10-18T17:54:22.501ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3932024-10-18T17:54:22.501ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3942024-10-18T17:54:22.501ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3952024-10-18T17:54:22.501ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3962024-10-18T17:54:22.537ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3972024-10-18T17:54:22.546ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e09cb849-5302-41dd-87a5-36e8723c48fb/crypt/zone file = sled-storage/src/dataset.rs:280
3982024-10-18T17:54:22.574ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/08066c3d-b7ed-49d2-8578-f8769fb94736) file = sled-agent/src/probe_manager.rs:230 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
3992024-10-18T17:54:22.717ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e09cb849-5302-41dd-87a5-36e8723c48fb file = sled-storage/src/dataset.rs:366
4002024-10-18T17:54:22.725ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
4012024-10-18T17:54:22.725ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
4022024-10-18T17:54:22.725ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4032024-10-18T17:54:22.725ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4042024-10-18T17:54:22.725ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4052024-10-18T17:54:22.735ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4062024-10-18T17:54:22.735ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool)
4072024-10-18T17:54:22.830ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }
4082024-10-18T17:54:22.838ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4092024-10-18T17:54:22.838ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4102024-10-18T17:54:22.838ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4112024-10-18T17:54:22.838ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4122024-10-18T17:54:22.838ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4132024-10-18T17:54:22.838ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4142024-10-18T17:54:22.838ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4152024-10-18T17:54:22.875ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4162024-10-18T17:54:22.885ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c6223f26-38f9-4483-9d17-efd8e60dd490/crypt/zone file = sled-storage/src/dataset.rs:280
4172024-10-18T17:54:23.058ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c6223f26-38f9-4483-9d17-efd8e60dd490 file = sled-storage/src/dataset.rs:366
4182024-10-18T17:54:23.066ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4192024-10-18T17:54:23.066ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4202024-10-18T17:54:23.066ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4212024-10-18T17:54:23.066ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4222024-10-18T17:54:23.066ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4232024-10-18T17:54:23.076ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4242024-10-18T17:54:23.076ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool)
4252024-10-18T17:54:23.187ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }
4262024-10-18T17:54:23.196ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4272024-10-18T17:54:23.196ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4282024-10-18T17:54:23.197ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4292024-10-18T17:54:23.197ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4302024-10-18T17:54:23.197ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4312024-10-18T17:54:23.197ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4322024-10-18T17:54:23.197ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4332024-10-18T17:54:23.234ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4342024-10-18T17:54:23.243ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_883d1cbf-e1f8-4c86-b18f-d626feb923fd/crypt/zone file = sled-storage/src/dataset.rs:280
4352024-10-18T17:54:23.420ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_883d1cbf-e1f8-4c86-b18f-d626feb923fd file = sled-storage/src/dataset.rs:366
4362024-10-18T17:54:23.428ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4372024-10-18T17:54:23.429ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4382024-10-18T17:54:23.429ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4392024-10-18T17:54:23.429ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e40afb95-7af8-4f62-aab4-e63b3bd02855 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 62298c00-fb90-4e92-a4b4-95bca961978c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4402024-10-18T17:54:23.429ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4412024-10-18T17:54:23.429ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4422024-10-18T17:54:23.429ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4432024-10-18T17:54:23.429ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:868 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4442024-10-18T17:54:23.429ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4452024-10-18T17:54:23.429ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:894 generation = Generation(1) sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4462024-10-18T17:54:23.429ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:899 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4472024-10-18T17:54:23.429ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:904 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4482024-10-18T17:54:23.429ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4492024-10-18T17:54:23.429ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:910 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4502024-10-18T17:54:23.429ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:798 gen = Generation(1)
4512024-10-18T17:54:23.429ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:915 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4522024-10-18T17:54:23.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4291616 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:60647 req_id = fb1884bc-ca5a-42b8-b0dd-14de6f052ddf response_code = 200 uri = /omicron-physical-disks
4532024-10-18T17:54:23.461ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4542024-10-18T17:54:23.461ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35122
4552024-10-18T17:54:23.461ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4562024-10-18T17:54:23.461ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
4572024-10-18T17:54:23.512ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4582024-10-18T17:54:23.512ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
4592024-10-18T17:54:23.564ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4602024-10-18T17:54:23.564ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4612024-10-18T17:54:23.576ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/08066c3d-b7ed-49d2-8578-f8769fb94736) file = sled-agent/src/probe_manager.rs:230 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4622024-10-18T17:54:23.616ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4632024-10-18T17:54:23.616ZINFOSledAgent (ServiceManager): No ledger in /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:188
4642024-10-18T17:54:23.616ZINFOSledAgent (ServiceManager): No ledger in /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:188
4652024-10-18T17:54:23.616ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4662024-10-18T17:54:23.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4672024-10-18T17:54:23.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4682024-10-18T17:54:23.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4692024-10-18T17:54:23.709ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 file = illumos-utils/src/zone.rs:316
4702024-10-18T17:54:23.713ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb file = illumos-utils/src/zone.rs:316
4712024-10-18T17:54:23.717ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6 file = illumos-utils/src/zone.rs:316
4722024-10-18T17:54:23.729ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 file = illumos-utils/src/zone.rs:354
4732024-10-18T17:54:23.734ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb file = illumos-utils/src/zone.rs:354
4742024-10-18T17:54:23.738ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6 file = illumos-utils/src/zone.rs:354
4752024-10-18T17:54:24.005ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 21.451816255s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4762024-10-18T17:54:24.579ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/08066c3d-b7ed-49d2-8578-f8769fb94736) file = sled-agent/src/probe_manager.rs:230 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
4772024-10-18T17:54:25.318ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4782024-10-18T17:54:25.318ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4792024-10-18T17:54:25.318ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
4802024-10-18T17:54:25.801ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4812024-10-18T17:54:26.281ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4822024-10-18T17:54:26.281ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4832024-10-18T17:54:26.281ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4842024-10-18T17:54:26.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 26.295178ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
4852024-10-18T17:54:26.289ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
4862024-10-18T17:54:26.293ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
4872024-10-18T17:54:26.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 142.471688ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
4882024-10-18T17:54:26.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 64.950725ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
4892024-10-18T17:54:26.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 69.929629ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
4902024-10-18T17:54:26.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 195.479ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
4912024-10-18T17:54:26.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 108.278131ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
4922024-10-18T17:54:26.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 75.939184ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
4932024-10-18T17:54:26.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 104.617399ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
4942024-10-18T17:54:26.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 276.739806ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
4952024-10-18T17:54:26.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 275.385509ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
4962024-10-18T17:54:26.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 207.928374ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
4972024-10-18T17:54:27.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 1.126623595s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
4982024-10-18T17:54:27.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 291.508488ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
4992024-10-18T17:54:27.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 736.265306ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
5002024-10-18T17:54:27.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 1.054917621s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
5012024-10-18T17:54:27.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 1.395360648s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
5022024-10-18T17:54:28.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 1.213849084s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
5032024-10-18T17:54:28.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 825.132846ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
5042024-10-18T17:54:28.567ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12885496655507240414 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
5052024-10-18T17:54:28.567ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6974983097420402160 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
5062024-10-18T17:54:28.952ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4269
5072024-10-18T17:54:28.952ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4279 gateway = fd00:1122:3344:101::1
5082024-10-18T17:54:28.958ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4358
5092024-10-18T17:54:29.061ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4406
5102024-10-18T17:54:29.061ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4433
5112024-10-18T17:54:29.099ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4446
5122024-10-18T17:54:29.099ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4310
5132024-10-18T17:54:29.151ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4352
5142024-10-18T17:54:29.151ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4413 rack_subnet = fd00:1122:3344::
5152024-10-18T17:54:29.170ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4424
5162024-10-18T17:54:29.170ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4466
5172024-10-18T17:54:29.238ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4498
5182024-10-18T17:54:29.239ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4504
5192024-10-18T17:54:29.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 1.402889407s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
5202024-10-18T17:54:29.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 1.220221456s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
5212024-10-18T17:54:29.342ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4545
5222024-10-18T17:54:29.342ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
5232024-10-18T17:54:29.353ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:53630
5242024-10-18T17:54:29.354ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:53630 req_id = dd3d1373-98bc-4650-9532-1f08ed96ecce response_code = 200 uri = /sled-identifiers
5252024-10-18T17:54:29.374ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:418
5262024-10-18T17:54:29.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 1.305274493s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
5272024-10-18T17:54:29.410ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:686
5282024-10-18T17:54:29.410ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:724
5292024-10-18T17:54:29.410ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:727 uptime = 0
5302024-10-18T17:54:29.410ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:440
5312024-10-18T17:54:29.448ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
5322024-10-18T17:54:29.495ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5332024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5342024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5352024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5362024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5372024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5382024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5392024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5402024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5412024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5422024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5432024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5442024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5452024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5462024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5472024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5482024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5492024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5502024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5512024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5522024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5532024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5542024-10-18T17:54:29.495ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5552024-10-18T17:54:30.264ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 291083675488840079 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
5562024-10-18T17:54:30.264ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7620151230303063215 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
5572024-10-18T17:54:30.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 579.381508ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
5582024-10-18T17:54:30.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb") failed: Property not found. retry in 1.38593567s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
5592024-10-18T17:54:30.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88") failed: Property not found. retry in 1.000925378s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
5602024-10-18T17:54:31.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6") failed: Property not found. retry in 606.64242ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
5612024-10-18T17:54:32.069ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_1fb67d4e-bac4-4e8f-b10a-4ef050bccb88
5622024-10-18T17:54:32.069ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_5a24845a-5b82-4ab6-b0ce-11d63d68b9cb
5632024-10-18T17:54:32.069ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_7a37cb7b-e967-4955-ad9c-4441ce0d32e6
5642024-10-18T17:54:32.069ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5652024-10-18T17:54:32.070ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5662024-10-18T17:54:32.070ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8609164 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:35122 req_id = 74e2ba67-328e-49b3-ac51-0a573edda02a response_code = 204 uri = /omicron-zones
5672024-10-18T17:54:32.072ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5682024-10-18T17:54:32.118ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5692024-10-18T17:54:32.170ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5702024-10-18T17:54:32.216ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5712024-10-18T17:54:32.216ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5722024-10-18T17:54:32.246ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5732024-10-18T17:54:32.247ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5742024-10-18T17:54:32.247ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5752024-10-18T17:54:32.277ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5762024-10-18T17:54:32.277ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50903
5772024-10-18T17:54:32.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5782024-10-18T17:54:32.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
5792024-10-18T17:54:32.294ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5802024-10-18T17:54:32.294ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
5812024-10-18T17:54:32.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5822024-10-18T17:54:32.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5832024-10-18T17:54:32.328ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5842024-10-18T17:54:32.328ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
5852024-10-18T17:54:32.328ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
5862024-10-18T17:54:32.329ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5872024-10-18T17:54:32.340ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5882024-10-18T17:54:32.371ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5892024-10-18T17:54:32.372ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5902024-10-18T17:54:32.372ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5912024-10-18T17:54:32.401ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 176, 131]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5922024-10-18T17:54:32.401ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = boundary_ntp
5932024-10-18T17:54:32.402ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5942024-10-18T17:54:32.402ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5952024-10-18T17:54:32.402ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5962024-10-18T17:54:32.402ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5972024-10-18T17:54:32.424ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_64822dce-d64a-403f-bf53-d84635424320 file = illumos-utils/src/zone.rs:316
5982024-10-18T17:54:32.441ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_64822dce-d64a-403f-bf53-d84635424320 file = illumos-utils/src/zone.rs:354
5992024-10-18T17:54:33.072ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_64822dce-d64a-403f-bf53-d84635424320: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6002024-10-18T17:54:33.072ZINFOSledAgent (ServiceManager): Booting oxz_ntp_64822dce-d64a-403f-bf53-d84635424320 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6012024-10-18T17:54:33.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 35.093971ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6022024-10-18T17:54:33.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 131.479404ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6032024-10-18T17:54:33.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 105.055617ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6042024-10-18T17:54:33.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 413.175276ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6052024-10-18T17:54:33.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 941.434568ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6062024-10-18T17:54:34.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 850.107974ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6072024-10-18T17:54:35.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 1.056227109s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6082024-10-18T17:54:36.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_64822dce-d64a-403f-bf53-d84635424320") failed: Property not found. retry in 1.353024173s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6092024-10-18T17:54:38.190ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_ntp_64822dce-d64a-403f-bf53-d84635424320
6102024-10-18T17:54:38.190ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6112024-10-18T17:54:38.190ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6122024-10-18T17:54:38.191ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 5914102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:50903 req_id = af767690-0c77-4765-ba85-f307c83670bf response_code = 204 uri = /omicron-zones
6132024-10-18T17:54:38.193ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6142024-10-18T17:54:38.224ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6152024-10-18T17:54:38.224ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44038
6162024-10-18T17:54:38.233ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_64822dce-d64a-403f-bf53-d84635424320': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6172024-10-18T17:54:38.233ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8872 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44038 req_id = 5a503c60-fcfa-4c28-b92c-b526987dea5e response_code = 500 uri = /timesync
6182024-10-18T17:54:38.236ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6192024-10-18T17:54:38.362ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6202024-10-18T17:54:38.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59081
6212024-10-18T17:54:42.371ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_64822dce-d64a-403f-bf53-d84635424320': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6222024-10-18T17:54:42.371ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4008648 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59081 req_id = bdcd1c3f-ec0c-4fd0-a040-ec4b5e431c92 response_code = 500 uri = /timesync
6232024-10-18T17:54:42.373ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4809265991385465253 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
6242024-10-18T17:54:42.373ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3909211028453507039 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
6252024-10-18T17:54:42.373ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11979277470628982865 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
6262024-10-18T17:54:42.374ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12885496655507240414 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
6272024-10-18T17:54:42.374ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6974983097420402160 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
6282024-10-18T17:54:42.374ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 291083675488840079 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
6292024-10-18T17:54:42.374ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7620151230303063215 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
6302024-10-18T17:54:42.375ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6312024-10-18T17:54:42.532ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6322024-10-18T17:54:42.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33386
6332024-10-18T17:54:42.540ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_64822dce-d64a-403f-bf53-d84635424320': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6342024-10-18T17:54:42.540ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8493 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33386 req_id = e453b4fe-88d8-4608-b09f-59af067e3a44 response_code = 500 uri = /timesync
6352024-10-18T17:54:42.543ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6362024-10-18T17:54:42.688ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6372024-10-18T17:54:42.688ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49074
6382024-10-18T17:54:46.002ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_64822dce-d64a-403f-bf53-d84635424320': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6392024-10-18T17:54:46.002ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4007865 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49074 req_id = fb3648cc-9b9e-415b-b9af-d47eafd27b46 response_code = 500 uri = /timesync
6402024-10-18T17:54:46.003ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6412024-10-18T17:54:46.280ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6422024-10-18T17:54:46.280ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39824
6432024-10-18T17:54:46.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 20409 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39824 req_id = 533a8015-407e-42dd-bd0f-0fde8922b732 response_code = 200 uri = /timesync
6442024-10-18T17:54:46.304ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1729274085.3730578, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6452024-10-18T17:54:46.304ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1298
6462024-10-18T17:54:46.335ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6472024-10-18T17:54:46.335ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38201
6482024-10-18T17:54:46.336ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6492024-10-18T17:54:46.336ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
6502024-10-18T17:54:46.355ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6512024-10-18T17:54:46.355ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
6522024-10-18T17:54:46.373ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6532024-10-18T17:54:46.373ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6542024-10-18T17:54:46.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6552024-10-18T17:54:46.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
6562024-10-18T17:54:46.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6572024-10-18T17:54:46.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
6582024-10-18T17:54:46.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6592024-10-18T17:54:46.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6602024-10-18T17:54:46.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6612024-10-18T17:54:46.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6622024-10-18T17:54:46.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6632024-10-18T17:54:46.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6642024-10-18T17:54:46.643ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6652024-10-18T17:54:46.643ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
6662024-10-18T17:54:46.643ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
6672024-10-18T17:54:46.643ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6682024-10-18T17:54:46.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
6692024-10-18T17:54:46.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
6702024-10-18T17:54:46.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
6712024-10-18T17:54:46.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
6722024-10-18T17:54:46.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
6732024-10-18T17:54:46.674ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03", TimeNotSynchronized), ("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb", TimeNotSynchronized), ("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34", TimeNotSynchronized), ("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d", TimeNotSynchronized), ("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 338478 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38201 req_id = c24ab5f5-b36b-4dee-9f53-60c5bfa29661 response_code = 500 uri = /omicron-zones
6742024-10-18T17:54:46.674ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c24ab5f5-b36b-4dee-9f53-60c5bfa29661", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c24ab5f5-b36b-4dee-9f53-60c5bfa29661" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 137.453533ms sled_address = [fd00:1122:3344:101::1]:12345
6752024-10-18T17:54:46.814ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6762024-10-18T17:54:46.814ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6772024-10-18T17:54:46.814ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
6782024-10-18T17:54:46.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6792024-10-18T17:54:46.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
6802024-10-18T17:54:46.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6812024-10-18T17:54:46.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6822024-10-18T17:54:46.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6832024-10-18T17:54:46.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
6842024-10-18T17:54:46.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6852024-10-18T17:54:46.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
6862024-10-18T17:54:46.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6872024-10-18T17:54:46.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6882024-10-18T17:54:46.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6892024-10-18T17:54:46.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6902024-10-18T17:54:46.932ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6912024-10-18T17:54:46.932ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6922024-10-18T17:54:46.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6932024-10-18T17:54:46.949ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
6942024-10-18T17:54:46.949ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
6952024-10-18T17:54:46.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6962024-10-18T17:54:46.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
6972024-10-18T17:54:46.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
6982024-10-18T17:54:46.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
6992024-10-18T17:54:46.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7002024-10-18T17:54:46.978ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7012024-10-18T17:54:46.978ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03", TimeNotSynchronized), ("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb", TimeNotSynchronized), ("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d", TimeNotSynchronized), ("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34", TimeNotSynchronized), ("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 164548 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38201 req_id = 53ef5d20-b5d8-4530-9f39-9368ae94d807 response_code = 500 uri = /omicron-zones
7022024-10-18T17:54:46.978ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "53ef5d20-b5d8-4530-9f39-9368ae94d807", "content-length": "124", "date": "Fri, 18 Oct 2024 17:54:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "53ef5d20-b5d8-4530-9f39-9368ae94d807" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 98.125762ms sled_address = [fd00:1122:3344:101::1]:12345
7032024-10-18T17:54:47.079ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7042024-10-18T17:54:47.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7052024-10-18T17:54:47.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
7062024-10-18T17:54:47.013ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7072024-10-18T17:54:47.013ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
7082024-10-18T17:54:47.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7092024-10-18T17:54:47.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7102024-10-18T17:54:47.048ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7112024-10-18T17:54:47.048ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
7122024-10-18T17:54:47.066ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7132024-10-18T17:54:47.066ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
7142024-10-18T17:54:47.084ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7152024-10-18T17:54:47.084ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7162024-10-18T17:54:47.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7172024-10-18T17:54:47.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7182024-10-18T17:54:47.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7192024-10-18T17:54:47.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7202024-10-18T17:54:47.138ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7212024-10-18T17:54:47.138ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
7222024-10-18T17:54:47.138ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
7232024-10-18T17:54:47.138ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7242024-10-18T17:54:47.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7252024-10-18T17:54:47.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7262024-10-18T17:54:47.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7272024-10-18T17:54:47.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7282024-10-18T17:54:47.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7292024-10-18T17:54:47.308ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34 file = illumos-utils/src/zone.rs:316
7302024-10-18T17:54:47.312ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb file = illumos-utils/src/zone.rs:316
7312024-10-18T17:54:47.316ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d file = illumos-utils/src/zone.rs:316
7322024-10-18T17:54:47.320ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03 file = illumos-utils/src/zone.rs:316
7332024-10-18T17:54:47.324ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a file = illumos-utils/src/zone.rs:316
7342024-10-18T17:54:47.329ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34 file = illumos-utils/src/zone.rs:354
7352024-10-18T17:54:47.333ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb file = illumos-utils/src/zone.rs:354
7362024-10-18T17:54:47.337ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d file = illumos-utils/src/zone.rs:354
7372024-10-18T17:54:47.342ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03 file = illumos-utils/src/zone.rs:354
7382024-10-18T17:54:47.346ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a file = illumos-utils/src/zone.rs:354
7392024-10-18T17:54:50.356ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='8e79611e-ba88-4a64-8bf6-28d7df64e66a'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7402024-10-18T17:54:50.356ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='69bfa368-1d69-4d23-88e9-3383e573f0eb'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7412024-10-18T17:54:50.356ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7422024-10-18T17:54:50.362ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7432024-10-18T17:54:50.374ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='2d1c58db-a735-48db-90ac-891756bdde5d'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7442024-10-18T17:54:50.374ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='03d0126b-4566-466f-bb9c-de9896115c34'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7452024-10-18T17:54:50.374ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7462024-10-18T17:54:50.379ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7472024-10-18T17:54:50.384ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='68422c74-fc22-434f-add8-652fc427fc03'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7482024-10-18T17:54:50.384ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7492024-10-18T17:54:50.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 49.301977ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7502024-10-18T17:54:50.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 38.733129ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7512024-10-18T17:54:50.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 42.748201ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7522024-10-18T17:54:50.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 72.430008ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7532024-10-18T17:54:50.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 46.031489ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7542024-10-18T17:54:50.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 59.188762ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7552024-10-18T17:54:50.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 114.465556ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7562024-10-18T17:54:50.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 119.558499ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7572024-10-18T17:54:50.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 247.651193ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7582024-10-18T17:54:50.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 110.83957ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7592024-10-18T17:54:50.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 78.32511ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7602024-10-18T17:54:51.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 243.629208ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7612024-10-18T17:54:51.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 114.209489ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7622024-10-18T17:54:51.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 443.78318ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7632024-10-18T17:54:51.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 214.568604ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7642024-10-18T17:54:51.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 210.278538ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7652024-10-18T17:54:52.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 492.942322ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7662024-10-18T17:54:52.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 570.977929ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7672024-10-18T17:54:52.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 980.113588ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7682024-10-18T17:54:52.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 553.802519ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7692024-10-18T17:54:52.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 579.724812ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7702024-10-18T17:54:52.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 738.398804ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7712024-10-18T17:54:52.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 635.75826ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7722024-10-18T17:54:52.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 1.057857144s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7732024-10-18T17:54:53.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 843.993059ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7742024-10-18T17:54:53.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 1.079107311s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7752024-10-18T17:54:53.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 1.36663636s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7762024-10-18T17:54:53.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 1.186310675s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7772024-10-18T17:54:54.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 1.13090862s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7782024-10-18T17:54:54.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 534.384077ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7792024-10-18T17:54:54.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 694.246759ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7802024-10-18T17:54:54.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 1.431726173s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7812024-10-18T17:54:54.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 712.404074ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7822024-10-18T17:54:54.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 775.971177ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7832024-10-18T17:54:55.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 1.246628232s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7842024-10-18T17:54:55.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 952.995578ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7852024-10-18T17:54:55.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 1.482880995s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7862024-10-18T17:54:55.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 1.254498093s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7872024-10-18T17:54:56.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 651.825913ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7882024-10-18T17:54:56.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 1.451862612s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7892024-10-18T17:54:56.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 1.294208726s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7902024-10-18T17:54:56.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 618.98188ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7912024-10-18T17:54:56.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 976.389981ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7922024-10-18T17:54:57.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 1.217010827s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7932024-10-18T17:54:57.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 785.704912ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7942024-10-18T17:54:57.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 1.191093613s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
7952024-10-18T17:54:57.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 1.483756319s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
7962024-10-18T17:54:57.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 549.280643ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
7972024-10-18T17:54:58.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 1.048211776s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
7982024-10-18T17:54:58.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 899.924064ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
7992024-10-18T17:54:58.407ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52904
8002024-10-18T17:54:58.407ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 77 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52904 req_id = 97d0c979-d9c8-44bb-aaa0-776fd406b01f response_code = 200 uri = /baseboard
8012024-10-18T17:54:58.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 636.409353ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8022024-10-18T17:54:58.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 891.624489ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
8032024-10-18T17:54:59.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 1.185864855s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8042024-10-18T17:54:59.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 625.398321ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
8052024-10-18T17:54:59.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 922.756867ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
8062024-10-18T17:54:59.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 973.485384ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
8072024-10-18T17:54:59.746ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 259.190851ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
8082024-10-18T17:54:59.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a") failed: Property not found. retry in 805.004322ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
8092024-10-18T17:54:59.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03") failed: Property not found. retry in 1.077882805s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
8102024-10-18T17:55:00.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d") failed: Property not found. retry in 550.321929ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
8112024-10-18T17:55:00.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb") failed: Property not found. retry in 624.659251ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
8122024-10-18T17:55:00.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 1.044973285s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8132024-10-18T17:55:01.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 503.461516ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8142024-10-18T17:55:01.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 885.483833ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8152024-10-18T17:55:02.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 974.183908ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8162024-10-18T17:55:04.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 656.834994ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8172024-10-18T17:55:04.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 1.087386371s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8182024-10-18T17:55:05.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 870.83681ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8192024-10-18T17:55:06.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 1.371036044s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8202024-10-18T17:55:08.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 624.177084ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8212024-10-18T17:55:08.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 917.36785ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8222024-10-18T17:55:09.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 588.229672ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8232024-10-18T17:55:10.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 779.095183ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8242024-10-18T17:55:11.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 539.821614ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8252024-10-18T17:55:11.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 930.596343ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8262024-10-18T17:55:12.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 1.219624207s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8272024-10-18T17:55:13.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34") failed: Property not found. retry in 755.250876ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8282024-10-18T17:55:14.557ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34
8292024-10-18T17:55:14.557ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_69bfa368-1d69-4d23-88e9-3383e573f0eb
8302024-10-18T17:55:14.557ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_2d1c58db-a735-48db-90ac-891756bdde5d
8312024-10-18T17:55:14.557ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_68422c74-fc22-434f-add8-652fc427fc03
8322024-10-18T17:55:14.557ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_8e79611e-ba88-4a64-8bf6-28d7df64e66a
8332024-10-18T17:55:14.557ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8342024-10-18T17:55:14.571ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8352024-10-18T17:55:14.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 27519125 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38201 req_id = 4b95b950-eb0f-4faf-b20e-6e396ba3929b response_code = 204 uri = /omicron-zones
8362024-10-18T17:55:14.617ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55162
8372024-10-18T17:55:14.651ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_03d0126b-4566-466f-bb9c-de9896115c34 /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Could not get address: Object not found file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 33682 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:55162 req_id = 5dfd9717-9bf1-47b3-9a1d-7e11cba304a7 response_code = 500 uri = /cockroachdb
8382024-10-18T17:55:14.651ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5dfd9717-9bf1-47b3-9a1d-7e11cba304a7", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5dfd9717-9bf1-47b3-9a1d-7e11cba304a7" } file = sled-agent/src/rack_setup/service.rs:1027 retry_after = 145.186178ms
8392024-10-18T17:55:14.824ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::3]:32221 file = sled-agent/src/services.rs:3720
8402024-10-18T17:55:19.506ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3738
8412024-10-18T17:55:28.555ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3761
8422024-10-18T17:55:28.555ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 13720249 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:55162 req_id = a0276493-eec0-4875-91f9-0ef2d845a498 response_code = 204 uri = /cockroachdb
8432024-10-18T17:55:28.556ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63276
8442024-10-18T17:55:28.556ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63276 req_id = 19511586-e7e5-4203-9795-9062bc1fca21 response_code = 200 uri = /baseboard
8452024-10-18T17:55:28.559ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 595.957572ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
8462024-10-18T17:55:28.593ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8472024-10-18T17:55:28.593ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34297
8482024-10-18T17:55:28.593ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8492024-10-18T17:55:28.593ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
8502024-10-18T17:55:28.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8512024-10-18T17:55:28.618ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
8522024-10-18T17:55:28.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8532024-10-18T17:55:28.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8542024-10-18T17:55:28.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8552024-10-18T17:55:28.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
8562024-10-18T17:55:28.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8572024-10-18T17:55:28.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
8582024-10-18T17:55:28.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8592024-10-18T17:55:28.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8602024-10-18T17:55:28.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8612024-10-18T17:55:28.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8622024-10-18T17:55:28.735ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8632024-10-18T17:55:28.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8642024-10-18T17:55:28.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8652024-10-18T17:55:28.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8662024-10-18T17:55:28.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8672024-10-18T17:55:28.881ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8682024-10-18T17:55:28.973ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
8692024-10-18T17:55:28.973ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8702024-10-18T17:55:29.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8712024-10-18T17:55:29.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
8722024-10-18T17:55:29.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8732024-10-18T17:55:29.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
8742024-10-18T17:55:29.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8752024-10-18T17:55:29.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
8762024-10-18T17:55:29.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8772024-10-18T17:55:29.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
8782024-10-18T17:55:29.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8792024-10-18T17:55:29.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
8802024-10-18T17:55:29.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8812024-10-18T17:55:29.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
8822024-10-18T17:55:29.641ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8832024-10-18T17:55:29.641ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
8842024-10-18T17:55:29.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8852024-10-18T17:55:29.730ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
8862024-10-18T17:55:29.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8872024-10-18T17:55:29.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
8882024-10-18T17:55:29.885ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8892024-10-18T17:55:29.885ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
8902024-10-18T17:55:29.885ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
8912024-10-18T17:55:29.885ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8922024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
8932024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
8942024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
8952024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
8962024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
8972024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
8982024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
8992024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
9002024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
9012024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
9022024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
9032024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
9042024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
9052024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
9062024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
9072024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
9082024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
9092024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
9102024-10-18T17:55:29.989ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
9112024-10-18T17:55:29.989ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 1392022 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 32e85499-da82-40a7-b25b-b9d1d5ae0f86 response_code = 500 uri = /omicron-zones
9122024-10-18T17:55:29.989ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "32e85499-da82-40a7-b25b-b9d1d5ae0f86", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:29 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "32e85499-da82-40a7-b25b-b9d1d5ae0f86" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 104.844746ms sled_address = [fd00:1122:3344:101::1]:12345
9132024-10-18T17:55:30.095ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9142024-10-18T17:55:30.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9152024-10-18T17:55:30.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
9162024-10-18T17:55:30.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9172024-10-18T17:55:30.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
9182024-10-18T17:55:30.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9192024-10-18T17:55:30.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9202024-10-18T17:55:30.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9212024-10-18T17:55:30.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
9222024-10-18T17:55:30.171ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9232024-10-18T17:55:30.171ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
9242024-10-18T17:55:30.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9252024-10-18T17:55:30.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9262024-10-18T17:55:30.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9272024-10-18T17:55:30.207ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9282024-10-18T17:55:30.224ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9292024-10-18T17:55:30.224ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9302024-10-18T17:55:30.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9312024-10-18T17:55:30.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9322024-10-18T17:55:30.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9332024-10-18T17:55:30.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9342024-10-18T17:55:30.280ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
9352024-10-18T17:55:30.280ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9362024-10-18T17:55:30.299ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9372024-10-18T17:55:30.299ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
9382024-10-18T17:55:30.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9392024-10-18T17:55:30.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
9402024-10-18T17:55:30.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9412024-10-18T17:55:30.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
9422024-10-18T17:55:30.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9432024-10-18T17:55:30.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
9442024-10-18T17:55:30.365ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9452024-10-18T17:55:30.365ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
9462024-10-18T17:55:30.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9472024-10-18T17:55:30.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
9482024-10-18T17:55:30.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9492024-10-18T17:55:30.403ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
9502024-10-18T17:55:30.422ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9512024-10-18T17:55:30.422ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
9522024-10-18T17:55:30.442ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9532024-10-18T17:55:30.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
9542024-10-18T17:55:30.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9552024-10-18T17:55:30.461ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
9562024-10-18T17:55:30.461ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
9572024-10-18T17:55:30.462ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9582024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
9592024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
9602024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
9612024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
9622024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
9632024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
9642024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
9652024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
9662024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
9672024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
9682024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
9692024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
9702024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
9712024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
9722024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
9732024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
9742024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
9752024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
9762024-10-18T17:55:30.551ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
9772024-10-18T17:55:30.551ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 454033 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 84220936-2770-4154-832c-bef04d60dd6f response_code = 500 uri = /omicron-zones
9782024-10-18T17:55:30.551ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "84220936-2770-4154-832c-bef04d60dd6f", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "84220936-2770-4154-832c-bef04d60dd6f" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 160.407869ms sled_address = [fd00:1122:3344:101::1]:12345
9792024-10-18T17:55:30.713ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9802024-10-18T17:55:30.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9812024-10-18T17:55:30.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
9822024-10-18T17:55:30.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9832024-10-18T17:55:30.734ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
9842024-10-18T17:55:30.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9852024-10-18T17:55:30.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9862024-10-18T17:55:30.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9872024-10-18T17:55:30.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
9882024-10-18T17:55:30.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9892024-10-18T17:55:30.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
9902024-10-18T17:55:30.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9912024-10-18T17:55:30.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9922024-10-18T17:55:30.823ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9932024-10-18T17:55:30.823ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9942024-10-18T17:55:30.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9952024-10-18T17:55:30.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9962024-10-18T17:55:30.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9972024-10-18T17:55:30.855ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9982024-10-18T17:55:30.875ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9992024-10-18T17:55:30.875ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10002024-10-18T17:55:30.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
10012024-10-18T17:55:30.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10022024-10-18T17:55:30.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10032024-10-18T17:55:30.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
10042024-10-18T17:55:30.932ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10052024-10-18T17:55:30.932ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
10062024-10-18T17:55:30.950ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10072024-10-18T17:55:30.950ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
10082024-10-18T17:55:30.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10092024-10-18T17:55:30.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10102024-10-18T17:55:30.989ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10112024-10-18T17:55:30.989ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
10122024-10-18T17:55:31.009ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10132024-10-18T17:55:31.009ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
10142024-10-18T17:55:31.032ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10152024-10-18T17:55:31.032ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
10162024-10-18T17:55:31.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10172024-10-18T17:55:31.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
10182024-10-18T17:55:31.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10192024-10-18T17:55:31.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
10202024-10-18T17:55:31.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10212024-10-18T17:55:31.089ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
10222024-10-18T17:55:31.089ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
10232024-10-18T17:55:31.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10242024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
10252024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
10262024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
10272024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
10282024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
10292024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
10302024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
10312024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
10322024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
10332024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
10342024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
10352024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
10362024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
10372024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
10382024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
10392024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
10402024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
10412024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
10422024-10-18T17:55:31.184ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
10432024-10-18T17:55:31.185ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 470214 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 7517b818-03ba-404e-8806-d21f28f93bd7 response_code = 500 uri = /omicron-zones
10442024-10-18T17:55:31.185ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7517b818-03ba-404e-8806-d21f28f93bd7", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:31 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7517b818-03ba-404e-8806-d21f28f93bd7" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 169.912786ms sled_address = [fd00:1122:3344:101::1]:12345
10452024-10-18T17:55:31.357ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
10462024-10-18T17:55:31.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10472024-10-18T17:55:31.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
10482024-10-18T17:55:31.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10492024-10-18T17:55:31.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
10502024-10-18T17:55:31.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10512024-10-18T17:55:31.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10522024-10-18T17:55:31.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10532024-10-18T17:55:31.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
10542024-10-18T17:55:31.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10552024-10-18T17:55:31.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
10562024-10-18T17:55:31.459ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10572024-10-18T17:55:31.459ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10582024-10-18T17:55:31.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10592024-10-18T17:55:31.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10602024-10-18T17:55:31.495ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10612024-10-18T17:55:31.495ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10622024-10-18T17:55:31.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10632024-10-18T17:55:31.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10642024-10-18T17:55:31.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10652024-10-18T17:55:31.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10662024-10-18T17:55:31.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
10672024-10-18T17:55:31.555ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10682024-10-18T17:55:31.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10692024-10-18T17:55:31.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
10702024-10-18T17:55:31.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10712024-10-18T17:55:31.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
10722024-10-18T17:55:31.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10732024-10-18T17:55:31.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
10742024-10-18T17:55:31.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10752024-10-18T17:55:31.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10762024-10-18T17:55:31.652ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10772024-10-18T17:55:31.652ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
10782024-10-18T17:55:31.673ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10792024-10-18T17:55:31.673ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
10802024-10-18T17:55:31.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10812024-10-18T17:55:31.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
10822024-10-18T17:55:31.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10832024-10-18T17:55:31.711ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
10842024-10-18T17:55:31.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10852024-10-18T17:55:31.729ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
10862024-10-18T17:55:31.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10872024-10-18T17:55:31.749ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
10882024-10-18T17:55:31.749ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
10892024-10-18T17:55:31.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10902024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
10912024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
10922024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
10932024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
10942024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
10952024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
10962024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
10972024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
10982024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
10992024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
11002024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
11012024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
11022024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
11032024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
11042024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
11052024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
11062024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
11072024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
11082024-10-18T17:55:31.842ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
11092024-10-18T17:55:31.843ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 484347 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = df06e809-89bd-4436-ae89-006942c3fbda response_code = 500 uri = /omicron-zones
11102024-10-18T17:55:31.843ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "df06e809-89bd-4436-ae89-006942c3fbda", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:31 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "df06e809-89bd-4436-ae89-006942c3fbda" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 250.177432ms sled_address = [fd00:1122:3344:101::1]:12345
11112024-10-18T17:55:32.095ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
11122024-10-18T17:55:32.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11132024-10-18T17:55:32.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
11142024-10-18T17:55:32.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11152024-10-18T17:55:32.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
11162024-10-18T17:55:32.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11172024-10-18T17:55:32.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11182024-10-18T17:55:32.157ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11192024-10-18T17:55:32.157ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
11202024-10-18T17:55:32.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11212024-10-18T17:55:32.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
11222024-10-18T17:55:32.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11232024-10-18T17:55:32.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11242024-10-18T17:55:32.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11252024-10-18T17:55:32.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11262024-10-18T17:55:32.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11272024-10-18T17:55:32.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11282024-10-18T17:55:32.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11292024-10-18T17:55:32.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11302024-10-18T17:55:32.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11312024-10-18T17:55:32.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11322024-10-18T17:55:32.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
11332024-10-18T17:55:32.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11342024-10-18T17:55:32.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11352024-10-18T17:55:32.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
11362024-10-18T17:55:32.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11372024-10-18T17:55:32.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
11382024-10-18T17:55:32.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11392024-10-18T17:55:32.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
11402024-10-18T17:55:32.362ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11412024-10-18T17:55:32.362ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
11422024-10-18T17:55:32.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11432024-10-18T17:55:32.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
11442024-10-18T17:55:32.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11452024-10-18T17:55:32.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
11462024-10-18T17:55:32.418ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11472024-10-18T17:55:32.418ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
11482024-10-18T17:55:32.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11492024-10-18T17:55:32.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
11502024-10-18T17:55:32.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11512024-10-18T17:55:32.454ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
11522024-10-18T17:55:32.473ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11532024-10-18T17:55:32.473ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
11542024-10-18T17:55:32.473ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
11552024-10-18T17:55:32.473ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11562024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
11572024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
11582024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
11592024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
11602024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
11612024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
11622024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
11632024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
11642024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
11652024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
11662024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
11672024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
11682024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
11692024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
11702024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
11712024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
11722024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
11732024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
11742024-10-18T17:55:32.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
11752024-10-18T17:55:32.564ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 468418 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 2def43aa-5532-41bd-94d0-3d0e1c4163fa response_code = 500 uri = /omicron-zones
11762024-10-18T17:55:32.565ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2def43aa-5532-41bd-94d0-3d0e1c4163fa", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2def43aa-5532-41bd-94d0-3d0e1c4163fa" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 155.074076ms sled_address = [fd00:1122:3344:101::1]:12345
11772024-10-18T17:55:32.721ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
11782024-10-18T17:55:32.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11792024-10-18T17:55:32.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
11802024-10-18T17:55:32.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11812024-10-18T17:55:32.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
11822024-10-18T17:55:32.762ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11832024-10-18T17:55:32.762ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11842024-10-18T17:55:32.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11852024-10-18T17:55:32.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
11862024-10-18T17:55:32.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11872024-10-18T17:55:32.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
11882024-10-18T17:55:32.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11892024-10-18T17:55:32.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11902024-10-18T17:55:32.837ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11912024-10-18T17:55:32.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11922024-10-18T17:55:32.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11932024-10-18T17:55:32.855ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11942024-10-18T17:55:32.874ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11952024-10-18T17:55:32.874ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11962024-10-18T17:55:32.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11972024-10-18T17:55:32.897ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11982024-10-18T17:55:32.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
11992024-10-18T17:55:32.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12002024-10-18T17:55:32.939ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12012024-10-18T17:55:32.939ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
12022024-10-18T17:55:32.960ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12032024-10-18T17:55:32.960ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
12042024-10-18T17:55:32.981ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12052024-10-18T17:55:32.981ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
12062024-10-18T17:55:33.000ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12072024-10-18T17:55:33.000ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12082024-10-18T17:55:33.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12092024-10-18T17:55:33.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
12102024-10-18T17:55:33.044ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12112024-10-18T17:55:33.044ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
12122024-10-18T17:55:33.066ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12132024-10-18T17:55:33.066ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
12142024-10-18T17:55:33.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12152024-10-18T17:55:33.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
12162024-10-18T17:55:33.105ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12172024-10-18T17:55:33.105ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
12182024-10-18T17:55:33.124ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12192024-10-18T17:55:33.124ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
12202024-10-18T17:55:33.124ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
12212024-10-18T17:55:33.124ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12222024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
12232024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
12242024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
12252024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
12262024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
12272024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
12282024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
12292024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
12302024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
12312024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
12322024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
12332024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
12342024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
12352024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
12362024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
12372024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
12382024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
12392024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
12402024-10-18T17:55:33.217ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
12412024-10-18T17:55:33.217ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 494320 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 35a38fd7-c471-4425-af67-fad9fe49b3a9 response_code = 500 uri = /omicron-zones
12422024-10-18T17:55:33.217ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "35a38fd7-c471-4425-af67-fad9fe49b3a9", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "35a38fd7-c471-4425-af67-fad9fe49b3a9" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 207.727569ms sled_address = [fd00:1122:3344:101::1]:12345
12432024-10-18T17:55:33.427ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
12442024-10-18T17:55:33.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12452024-10-18T17:55:33.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
12462024-10-18T17:55:33.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12472024-10-18T17:55:33.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
12482024-10-18T17:55:33.466ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12492024-10-18T17:55:33.466ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12502024-10-18T17:55:33.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12512024-10-18T17:55:33.486ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
12522024-10-18T17:55:33.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12532024-10-18T17:55:33.504ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
12542024-10-18T17:55:33.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12552024-10-18T17:55:33.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12562024-10-18T17:55:33.541ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12572024-10-18T17:55:33.541ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12582024-10-18T17:55:33.560ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12592024-10-18T17:55:33.560ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12602024-10-18T17:55:33.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12612024-10-18T17:55:33.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12622024-10-18T17:55:33.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12632024-10-18T17:55:33.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12642024-10-18T17:55:33.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
12652024-10-18T17:55:33.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12662024-10-18T17:55:33.638ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12672024-10-18T17:55:33.638ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
12682024-10-18T17:55:33.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12692024-10-18T17:55:33.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
12702024-10-18T17:55:33.678ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12712024-10-18T17:55:33.678ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
12722024-10-18T17:55:33.697ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12732024-10-18T17:55:33.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12742024-10-18T17:55:33.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12752024-10-18T17:55:33.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
12762024-10-18T17:55:33.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12772024-10-18T17:55:33.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
12782024-10-18T17:55:33.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12792024-10-18T17:55:33.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
12802024-10-18T17:55:33.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12812024-10-18T17:55:33.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
12822024-10-18T17:55:33.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
12832024-10-18T17:55:33.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
12842024-10-18T17:55:33.814ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12852024-10-18T17:55:33.815ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
12862024-10-18T17:55:33.815ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
12872024-10-18T17:55:33.815ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12882024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
12892024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
12902024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
12912024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
12922024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
12932024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
12942024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
12952024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
12962024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
12972024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
12982024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
12992024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
13002024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
13012024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
13022024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
13032024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
13042024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
13052024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
13062024-10-18T17:55:33.906ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
13072024-10-18T17:55:33.906ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 477520 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = d54d10e4-204d-418b-b524-805f8fadd978 response_code = 500 uri = /omicron-zones
13082024-10-18T17:55:33.907ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d54d10e4-204d-418b-b524-805f8fadd978", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d54d10e4-204d-418b-b524-805f8fadd978" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 367.437872ms sled_address = [fd00:1122:3344:101::1]:12345
13092024-10-18T17:55:34.276ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
13102024-10-18T17:55:34.276ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13112024-10-18T17:55:34.276ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
13122024-10-18T17:55:34.299ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13132024-10-18T17:55:34.299ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
13142024-10-18T17:55:34.317ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13152024-10-18T17:55:34.317ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13162024-10-18T17:55:34.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13172024-10-18T17:55:34.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
13182024-10-18T17:55:34.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13192024-10-18T17:55:34.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
13202024-10-18T17:55:34.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13212024-10-18T17:55:34.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13222024-10-18T17:55:34.392ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13232024-10-18T17:55:34.392ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13242024-10-18T17:55:34.413ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13252024-10-18T17:55:34.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13262024-10-18T17:55:34.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13272024-10-18T17:55:34.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13282024-10-18T17:55:34.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13292024-10-18T17:55:34.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13302024-10-18T17:55:34.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
13312024-10-18T17:55:34.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13322024-10-18T17:55:34.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13332024-10-18T17:55:34.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
13342024-10-18T17:55:34.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13352024-10-18T17:55:34.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
13362024-10-18T17:55:34.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13372024-10-18T17:55:34.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
13382024-10-18T17:55:34.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13392024-10-18T17:55:34.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
13402024-10-18T17:55:34.562ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13412024-10-18T17:55:34.562ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
13422024-10-18T17:55:34.580ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13432024-10-18T17:55:34.580ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
13442024-10-18T17:55:34.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13452024-10-18T17:55:34.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
13462024-10-18T17:55:34.617ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13472024-10-18T17:55:34.617ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
13482024-10-18T17:55:34.634ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13492024-10-18T17:55:34.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
13502024-10-18T17:55:34.653ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13512024-10-18T17:55:34.653ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
13522024-10-18T17:55:34.653ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
13532024-10-18T17:55:34.653ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13542024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
13552024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
13562024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
13572024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
13582024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
13592024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
13602024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
13612024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
13622024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
13632024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
13642024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
13652024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
13662024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
13672024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
13682024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
13692024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
13702024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
13712024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
13722024-10-18T17:55:34.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
13732024-10-18T17:55:34.747ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 469492 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = fbce12b8-544b-4a2b-b228-747fcad8a807 response_code = 500 uri = /omicron-zones
13742024-10-18T17:55:34.747ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fbce12b8-544b-4a2b-b228-747fcad8a807", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fbce12b8-544b-4a2b-b228-747fcad8a807" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 297.053905ms sled_address = [fd00:1122:3344:101::1]:12345
13752024-10-18T17:55:35.047ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
13762024-10-18T17:55:35.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13772024-10-18T17:55:35.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
13782024-10-18T17:55:35.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13792024-10-18T17:55:35.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
13802024-10-18T17:55:35.090ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13812024-10-18T17:55:35.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13822024-10-18T17:55:35.109ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13832024-10-18T17:55:35.109ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
13842024-10-18T17:55:35.129ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13852024-10-18T17:55:35.129ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
13862024-10-18T17:55:35.149ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13872024-10-18T17:55:35.149ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13882024-10-18T17:55:35.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13892024-10-18T17:55:35.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13902024-10-18T17:55:35.186ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13912024-10-18T17:55:35.186ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13922024-10-18T17:55:35.205ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13932024-10-18T17:55:35.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13942024-10-18T17:55:35.225ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13952024-10-18T17:55:35.225ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13962024-10-18T17:55:35.243ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
13972024-10-18T17:55:35.243ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13982024-10-18T17:55:35.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
13992024-10-18T17:55:35.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
14002024-10-18T17:55:35.279ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14012024-10-18T17:55:35.279ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
14022024-10-18T17:55:35.298ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14032024-10-18T17:55:35.298ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
14042024-10-18T17:55:35.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14052024-10-18T17:55:35.316ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14062024-10-18T17:55:35.334ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14072024-10-18T17:55:35.334ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
14082024-10-18T17:55:35.352ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14092024-10-18T17:55:35.352ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
14102024-10-18T17:55:35.370ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14112024-10-18T17:55:35.370ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
14122024-10-18T17:55:35.388ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14132024-10-18T17:55:35.388ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
14142024-10-18T17:55:35.407ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14152024-10-18T17:55:35.407ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
14162024-10-18T17:55:35.425ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14172024-10-18T17:55:35.425ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
14182024-10-18T17:55:35.425ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
14192024-10-18T17:55:35.425ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14202024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
14212024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
14222024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
14232024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
14242024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
14252024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
14262024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
14272024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
14282024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
14292024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
14302024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
14312024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
14322024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
14332024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
14342024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
14352024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
14362024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
14372024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
14382024-10-18T17:55:35.524ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
14392024-10-18T17:55:35.524ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 476013 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 22852799-b765-47b6-ae9b-4601cae7abed response_code = 500 uri = /omicron-zones
14402024-10-18T17:55:35.524ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "22852799-b765-47b6-ae9b-4601cae7abed", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "22852799-b765-47b6-ae9b-4601cae7abed" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 262.637139ms sled_address = [fd00:1122:3344:101::1]:12345
14412024-10-18T17:55:35.788ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
14422024-10-18T17:55:35.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14432024-10-18T17:55:35.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
14442024-10-18T17:55:35.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14452024-10-18T17:55:35.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
14462024-10-18T17:55:35.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14472024-10-18T17:55:35.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14482024-10-18T17:55:35.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14492024-10-18T17:55:35.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
14502024-10-18T17:55:35.874ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14512024-10-18T17:55:35.874ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
14522024-10-18T17:55:35.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14532024-10-18T17:55:35.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14542024-10-18T17:55:35.912ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14552024-10-18T17:55:35.912ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14562024-10-18T17:55:35.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14572024-10-18T17:55:35.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14582024-10-18T17:55:35.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14592024-10-18T17:55:35.948ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14602024-10-18T17:55:35.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14612024-10-18T17:55:35.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14622024-10-18T17:55:35.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
14632024-10-18T17:55:35.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14642024-10-18T17:55:36.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14652024-10-18T17:55:36.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
14662024-10-18T17:55:36.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14672024-10-18T17:55:36.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
14682024-10-18T17:55:36.051ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14692024-10-18T17:55:36.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
14702024-10-18T17:55:36.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14712024-10-18T17:55:36.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14722024-10-18T17:55:36.092ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14732024-10-18T17:55:36.092ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
14742024-10-18T17:55:36.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14752024-10-18T17:55:36.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
14762024-10-18T17:55:36.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14772024-10-18T17:55:36.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
14782024-10-18T17:55:36.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14792024-10-18T17:55:36.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
14802024-10-18T17:55:36.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14812024-10-18T17:55:36.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
14822024-10-18T17:55:36.189ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14832024-10-18T17:55:36.189ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
14842024-10-18T17:55:36.189ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
14852024-10-18T17:55:36.189ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14862024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
14872024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
14882024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
14892024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
14902024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
14912024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
14922024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
14932024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
14942024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
14952024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
14962024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
14972024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
14982024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
14992024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
15002024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
15012024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
15022024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
15032024-10-18T17:55:36.282ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
15042024-10-18T17:55:36.283ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
15052024-10-18T17:55:36.283ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 492780 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = ad5e8e2b-369d-4585-99eb-cde017a68f24 response_code = 500 uri = /omicron-zones
15062024-10-18T17:55:36.283ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ad5e8e2b-369d-4585-99eb-cde017a68f24", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ad5e8e2b-369d-4585-99eb-cde017a68f24" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 412.15885ms sled_address = [fd00:1122:3344:101::1]:12345
15072024-10-18T17:55:36.698ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
15082024-10-18T17:55:36.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15092024-10-18T17:55:36.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
15102024-10-18T17:55:36.721ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15112024-10-18T17:55:36.721ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
15122024-10-18T17:55:36.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15132024-10-18T17:55:36.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15142024-10-18T17:55:36.760ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15152024-10-18T17:55:36.760ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
15162024-10-18T17:55:36.778ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15172024-10-18T17:55:36.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
15182024-10-18T17:55:36.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15192024-10-18T17:55:36.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15202024-10-18T17:55:36.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15212024-10-18T17:55:36.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15222024-10-18T17:55:36.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15232024-10-18T17:55:36.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15242024-10-18T17:55:36.851ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15252024-10-18T17:55:36.851ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15262024-10-18T17:55:36.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15272024-10-18T17:55:36.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15282024-10-18T17:55:36.891ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
15292024-10-18T17:55:36.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15302024-10-18T17:55:36.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15312024-10-18T17:55:36.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
15322024-10-18T17:55:36.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15332024-10-18T17:55:36.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
15342024-10-18T17:55:36.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15352024-10-18T17:55:36.948ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
15362024-10-18T17:55:36.966ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15372024-10-18T17:55:36.966ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
15382024-10-18T17:55:36.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15392024-10-18T17:55:36.984ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
15402024-10-18T17:55:37.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15412024-10-18T17:55:37.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
15422024-10-18T17:55:37.025ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15432024-10-18T17:55:37.025ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
15442024-10-18T17:55:37.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15452024-10-18T17:55:37.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
15462024-10-18T17:55:37.062ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15472024-10-18T17:55:37.062ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
15482024-10-18T17:55:37.082ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15492024-10-18T17:55:37.082ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
15502024-10-18T17:55:37.082ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
15512024-10-18T17:55:37.082ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15522024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
15532024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
15542024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
15552024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
15562024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
15572024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
15582024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
15592024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
15602024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
15612024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
15622024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
15632024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
15642024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
15652024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
15662024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
15672024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
15682024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
15692024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
15702024-10-18T17:55:37.177ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
15712024-10-18T17:55:37.177ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 478299 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = d72fc2c9-7686-42d5-9169-51c9aa9169bf response_code = 500 uri = /omicron-zones
15722024-10-18T17:55:37.177ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d72fc2c9-7686-42d5-9169-51c9aa9169bf", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d72fc2c9-7686-42d5-9169-51c9aa9169bf" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 666.609896ms sled_address = [fd00:1122:3344:101::1]:12345
15732024-10-18T17:55:37.847ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
15742024-10-18T17:55:37.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15752024-10-18T17:55:37.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
15762024-10-18T17:55:37.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15772024-10-18T17:55:37.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
15782024-10-18T17:55:37.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15792024-10-18T17:55:37.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15802024-10-18T17:55:37.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15812024-10-18T17:55:37.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
15822024-10-18T17:55:37.936ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15832024-10-18T17:55:37.936ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
15842024-10-18T17:55:37.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15852024-10-18T17:55:37.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15862024-10-18T17:55:37.975ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15872024-10-18T17:55:37.975ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15882024-10-18T17:55:37.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15892024-10-18T17:55:37.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15902024-10-18T17:55:38.014ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15912024-10-18T17:55:38.014ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15922024-10-18T17:55:38.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15932024-10-18T17:55:38.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15942024-10-18T17:55:38.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
15952024-10-18T17:55:38.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15962024-10-18T17:55:38.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15972024-10-18T17:55:38.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
15982024-10-18T17:55:38.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15992024-10-18T17:55:38.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
16002024-10-18T17:55:38.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16012024-10-18T17:55:38.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
16022024-10-18T17:55:38.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16032024-10-18T17:55:38.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16042024-10-18T17:55:38.152ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16052024-10-18T17:55:38.152ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
16062024-10-18T17:55:38.171ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16072024-10-18T17:55:38.171ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
16082024-10-18T17:55:38.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16092024-10-18T17:55:38.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
16102024-10-18T17:55:38.208ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16112024-10-18T17:55:38.208ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
16122024-10-18T17:55:38.227ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16132024-10-18T17:55:38.227ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
16142024-10-18T17:55:38.246ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16152024-10-18T17:55:38.247ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
16162024-10-18T17:55:38.247ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
16172024-10-18T17:55:38.247ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16182024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
16192024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
16202024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
16212024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
16222024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
16232024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
16242024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
16252024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
16262024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
16272024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
16282024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
16292024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
16302024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
16312024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
16322024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
16332024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
16342024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
16352024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
16362024-10-18T17:55:38.342ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
16372024-10-18T17:55:38.342ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 493472 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 9330f9bb-b40a-47af-85cc-6d25243ec9db response_code = 500 uri = /omicron-zones
16382024-10-18T17:55:38.342ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9330f9bb-b40a-47af-85cc-6d25243ec9db", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9330f9bb-b40a-47af-85cc-6d25243ec9db" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 653.731712ms sled_address = [fd00:1122:3344:101::1]:12345
16392024-10-18T17:55:38.999ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
16402024-10-18T17:55:38.999ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
16412024-10-18T17:55:38.999ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
16422024-10-18T17:55:39.020ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
16432024-10-18T17:55:39.020ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
16442024-10-18T17:55:39.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
16452024-10-18T17:55:39.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16462024-10-18T17:55:39.061ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
16472024-10-18T17:55:39.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
16482024-10-18T17:55:39.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
16492024-10-18T17:55:39.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
16502024-10-18T17:55:39.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
16512024-10-18T17:55:39.097ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
16522024-10-18T17:55:39.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
16532024-10-18T17:55:39.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
16542024-10-18T17:55:39.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
16552024-10-18T17:55:39.131ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
16562024-10-18T17:55:39.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
16572024-10-18T17:55:39.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16582024-10-18T17:55:39.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
16592024-10-18T17:55:39.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16602024-10-18T17:55:39.188ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
16612024-10-18T17:55:39.188ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16622024-10-18T17:55:39.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16632024-10-18T17:55:39.207ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
16642024-10-18T17:55:39.226ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16652024-10-18T17:55:39.226ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
16662024-10-18T17:55:39.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16672024-10-18T17:55:39.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
16682024-10-18T17:55:39.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16692024-10-18T17:55:39.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16702024-10-18T17:55:39.280ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16712024-10-18T17:55:39.280ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
16722024-10-18T17:55:39.296ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16732024-10-18T17:55:39.296ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
16742024-10-18T17:55:39.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16752024-10-18T17:55:39.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
16762024-10-18T17:55:39.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16772024-10-18T17:55:39.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
16782024-10-18T17:55:39.350ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
16792024-10-18T17:55:39.350ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
16802024-10-18T17:55:39.368ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16812024-10-18T17:55:39.368ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
16822024-10-18T17:55:39.368ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
16832024-10-18T17:55:39.368ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16842024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
16852024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
16862024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
16872024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
16882024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
16892024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
16902024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
16912024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
16922024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
16932024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
16942024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
16952024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
16962024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
16972024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
16982024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
16992024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
17002024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
17012024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
17022024-10-18T17:55:39.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
17032024-10-18T17:55:39.459ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 459175 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 29e1d1b7-e539-48d2-a1a9-1152f0984ad7 response_code = 500 uri = /omicron-zones
17042024-10-18T17:55:39.459ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "29e1d1b7-e539-48d2-a1a9-1152f0984ad7", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:38 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "29e1d1b7-e539-48d2-a1a9-1152f0984ad7" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 798.541154ms sled_address = [fd00:1122:3344:101::1]:12345
17052024-10-18T17:55:40.261ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
17062024-10-18T17:55:40.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17072024-10-18T17:55:40.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
17082024-10-18T17:55:40.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17092024-10-18T17:55:40.283ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
17102024-10-18T17:55:40.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17112024-10-18T17:55:40.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17122024-10-18T17:55:40.320ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17132024-10-18T17:55:40.320ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
17142024-10-18T17:55:40.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17152024-10-18T17:55:40.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
17162024-10-18T17:55:40.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17172024-10-18T17:55:40.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17182024-10-18T17:55:40.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17192024-10-18T17:55:40.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17202024-10-18T17:55:40.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17212024-10-18T17:55:40.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17222024-10-18T17:55:40.413ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17232024-10-18T17:55:40.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17242024-10-18T17:55:40.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17252024-10-18T17:55:40.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17262024-10-18T17:55:40.452ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
17272024-10-18T17:55:40.452ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17282024-10-18T17:55:40.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17292024-10-18T17:55:40.471ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
17302024-10-18T17:55:40.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17312024-10-18T17:55:40.486ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
17322024-10-18T17:55:40.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17332024-10-18T17:55:40.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
17342024-10-18T17:55:40.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17352024-10-18T17:55:40.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
17362024-10-18T17:55:40.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17372024-10-18T17:55:40.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
17382024-10-18T17:55:40.562ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17392024-10-18T17:55:40.562ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
17402024-10-18T17:55:40.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17412024-10-18T17:55:40.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
17422024-10-18T17:55:40.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17432024-10-18T17:55:40.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
17442024-10-18T17:55:40.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17452024-10-18T17:55:40.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
17462024-10-18T17:55:40.640ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17472024-10-18T17:55:40.640ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
17482024-10-18T17:55:40.640ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
17492024-10-18T17:55:40.640ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17502024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
17512024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
17522024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
17532024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
17542024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
17552024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
17562024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
17572024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
17582024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
17592024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
17602024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
17612024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
17622024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
17632024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
17642024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
17652024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
17662024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
17672024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
17682024-10-18T17:55:40.730ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
17692024-10-18T17:55:40.731ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 467845 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = fc7f4fa0-1ecb-4d73-9b33-62f4a0054b35 response_code = 500 uri = /omicron-zones
17702024-10-18T17:55:40.731ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fc7f4fa0-1ecb-4d73-9b33-62f4a0054b35", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:40 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fc7f4fa0-1ecb-4d73-9b33-62f4a0054b35" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 1.565844505s sled_address = [fd00:1122:3344:101::1]:12345
17712024-10-18T17:55:42.302ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
17722024-10-18T17:55:42.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17732024-10-18T17:55:42.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
17742024-10-18T17:55:42.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17752024-10-18T17:55:42.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
17762024-10-18T17:55:42.345ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17772024-10-18T17:55:42.345ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17782024-10-18T17:55:42.365ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17792024-10-18T17:55:42.365ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
17802024-10-18T17:55:42.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17812024-10-18T17:55:42.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
17822024-10-18T17:55:42.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17832024-10-18T17:55:42.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17842024-10-18T17:55:42.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17852024-10-18T17:55:42.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17862024-10-18T17:55:42.443ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17872024-10-18T17:55:42.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17882024-10-18T17:55:42.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17892024-10-18T17:55:42.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17902024-10-18T17:55:42.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17912024-10-18T17:55:42.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17922024-10-18T17:55:42.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
17932024-10-18T17:55:42.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17942024-10-18T17:55:42.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17952024-10-18T17:55:42.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
17962024-10-18T17:55:42.539ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17972024-10-18T17:55:42.539ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
17982024-10-18T17:55:42.557ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17992024-10-18T17:55:42.557ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
18002024-10-18T17:55:42.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18012024-10-18T17:55:42.575ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18022024-10-18T17:55:42.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18032024-10-18T17:55:42.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
18042024-10-18T17:55:42.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18052024-10-18T17:55:42.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
18062024-10-18T17:55:42.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18072024-10-18T17:55:42.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
18082024-10-18T17:55:42.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18092024-10-18T17:55:42.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
18102024-10-18T17:55:42.673ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18112024-10-18T17:55:42.673ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
18122024-10-18T17:55:42.692ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18132024-10-18T17:55:42.693ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
18142024-10-18T17:55:42.693ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
18152024-10-18T17:55:42.693ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18162024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
18172024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
18182024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
18192024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
18202024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
18212024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
18222024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
18232024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
18242024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
18252024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
18262024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
18272024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
18282024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
18292024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
18302024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
18312024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
18322024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
18332024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
18342024-10-18T17:55:42.788ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
18352024-10-18T17:55:42.788ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 484880 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 1f6ca58e-fa0d-47f4-af0a-3de6c1736530 response_code = 500 uri = /omicron-zones
18362024-10-18T17:55:42.788ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1f6ca58e-fa0d-47f4-af0a-3de6c1736530", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:42 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1f6ca58e-fa0d-47f4-af0a-3de6c1736530" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 1.641702666s sled_address = [fd00:1122:3344:101::1]:12345
18372024-10-18T17:55:44.238ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.4065373s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
18382024-10-18T17:55:44.437ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
18392024-10-18T17:55:44.437ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
18402024-10-18T17:55:44.437ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
18412024-10-18T17:55:44.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
18422024-10-18T17:55:44.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
18432024-10-18T17:55:44.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
18442024-10-18T17:55:44.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18452024-10-18T17:55:44.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
18462024-10-18T17:55:44.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
18472024-10-18T17:55:44.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
18482024-10-18T17:55:44.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
18492024-10-18T17:55:44.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
18502024-10-18T17:55:44.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
18512024-10-18T17:55:44.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
18522024-10-18T17:55:44.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
18532024-10-18T17:55:44.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
18542024-10-18T17:55:44.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
18552024-10-18T17:55:44.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
18562024-10-18T17:55:44.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18572024-10-18T17:55:44.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
18582024-10-18T17:55:44.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18592024-10-18T17:55:44.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
18602024-10-18T17:55:44.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18612024-10-18T17:55:44.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18622024-10-18T17:55:44.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
18632024-10-18T17:55:44.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18642024-10-18T17:55:44.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
18652024-10-18T17:55:44.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18662024-10-18T17:55:44.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
18672024-10-18T17:55:44.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18682024-10-18T17:55:44.718ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18692024-10-18T17:55:44.739ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18702024-10-18T17:55:44.739ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
18712024-10-18T17:55:44.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18722024-10-18T17:55:44.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
18732024-10-18T17:55:44.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18742024-10-18T17:55:44.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
18752024-10-18T17:55:44.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18762024-10-18T17:55:44.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
18772024-10-18T17:55:44.815ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
18782024-10-18T17:55:44.815ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
18792024-10-18T17:55:44.836ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18802024-10-18T17:55:44.836ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
18812024-10-18T17:55:44.837ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
18822024-10-18T17:55:44.837ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18832024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
18842024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
18852024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
18862024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
18872024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
18882024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
18892024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
18902024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
18912024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
18922024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
18932024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
18942024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
18952024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
18962024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
18972024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
18982024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
18992024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
19002024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
19012024-10-18T17:55:44.936ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
19022024-10-18T17:55:44.936ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 498124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 90be6c4f-4f95-40b5-b72e-189d38da9665 response_code = 500 uri = /omicron-zones
19032024-10-18T17:55:44.936ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "90be6c4f-4f95-40b5-b72e-189d38da9665", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "90be6c4f-4f95-40b5-b72e-189d38da9665" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 908.765523ms sled_address = [fd00:1122:3344:101::1]:12345
19042024-10-18T17:55:45.848ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
19052024-10-18T17:55:45.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19062024-10-18T17:55:45.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
19072024-10-18T17:55:45.870ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19082024-10-18T17:55:45.870ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
19092024-10-18T17:55:45.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19102024-10-18T17:55:45.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19112024-10-18T17:55:45.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19122024-10-18T17:55:45.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
19132024-10-18T17:55:45.927ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19142024-10-18T17:55:45.927ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
19152024-10-18T17:55:45.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19162024-10-18T17:55:45.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19172024-10-18T17:55:45.965ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19182024-10-18T17:55:45.965ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19192024-10-18T17:55:45.986ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19202024-10-18T17:55:45.986ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19212024-10-18T17:55:46.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19222024-10-18T17:55:46.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19232024-10-18T17:55:46.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19242024-10-18T17:55:46.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19252024-10-18T17:55:46.048ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
19262024-10-18T17:55:46.048ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19272024-10-18T17:55:46.067ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19282024-10-18T17:55:46.067ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
19292024-10-18T17:55:46.087ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19302024-10-18T17:55:46.087ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
19312024-10-18T17:55:46.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19322024-10-18T17:55:46.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
19332024-10-18T17:55:46.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19342024-10-18T17:55:46.126ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
19352024-10-18T17:55:46.145ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19362024-10-18T17:55:46.145ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
19372024-10-18T17:55:46.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19382024-10-18T17:55:46.164ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
19392024-10-18T17:55:46.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19402024-10-18T17:55:46.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
19412024-10-18T17:55:46.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19422024-10-18T17:55:46.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
19432024-10-18T17:55:46.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19442024-10-18T17:55:46.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
19452024-10-18T17:55:46.241ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
19462024-10-18T17:55:46.242ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
19472024-10-18T17:55:46.242ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
19482024-10-18T17:55:46.242ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
19492024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
19502024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
19512024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
19522024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
19532024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
19542024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
19552024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
19562024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
19572024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
19582024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
19592024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
19602024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
19612024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
19622024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
19632024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
19642024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
19652024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
19662024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
19672024-10-18T17:55:46.338ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
19682024-10-18T17:55:46.338ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 487912 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = c8a2767d-89f9-45d2-8eb3-40419353b1fe response_code = 500 uri = /omicron-zones
19692024-10-18T17:55:46.338ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c8a2767d-89f9-45d2-8eb3-40419353b1fe", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c8a2767d-89f9-45d2-8eb3-40419353b1fe" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 1.102977596s sled_address = [fd00:1122:3344:101::1]:12345
19702024-10-18T17:55:47.445ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
19712024-10-18T17:55:47.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19722024-10-18T17:55:47.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
19732024-10-18T17:55:47.467ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19742024-10-18T17:55:47.467ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
19752024-10-18T17:55:47.487ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19762024-10-18T17:55:47.487ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19772024-10-18T17:55:47.506ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19782024-10-18T17:55:47.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
19792024-10-18T17:55:47.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19802024-10-18T17:55:47.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
19812024-10-18T17:55:47.545ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19822024-10-18T17:55:47.545ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19832024-10-18T17:55:47.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19842024-10-18T17:55:47.563ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19852024-10-18T17:55:47.580ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
19862024-10-18T17:55:47.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19872024-10-18T17:55:47.600ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19882024-10-18T17:55:47.600ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19892024-10-18T17:55:47.620ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
19902024-10-18T17:55:47.620ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19912024-10-18T17:55:47.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
19922024-10-18T17:55:47.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19932024-10-18T17:55:47.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19942024-10-18T17:55:47.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
19952024-10-18T17:55:47.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19962024-10-18T17:55:47.676ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
19972024-10-18T17:55:47.695ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
19982024-10-18T17:55:47.695ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
19992024-10-18T17:55:47.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20002024-10-18T17:55:47.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
20012024-10-18T17:55:47.731ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20022024-10-18T17:55:47.731ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
20032024-10-18T17:55:47.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20042024-10-18T17:55:47.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
20052024-10-18T17:55:47.768ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20062024-10-18T17:55:47.768ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
20072024-10-18T17:55:47.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20082024-10-18T17:55:47.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
20092024-10-18T17:55:47.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20102024-10-18T17:55:47.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
20112024-10-18T17:55:47.825ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20122024-10-18T17:55:47.825ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
20132024-10-18T17:55:47.826ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
20142024-10-18T17:55:47.826ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20152024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
20162024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
20172024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
20182024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
20192024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
20202024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
20212024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
20222024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
20232024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
20242024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
20252024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
20262024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
20272024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
20282024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
20292024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
20302024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
20312024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
20322024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
20332024-10-18T17:55:47.921ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
20342024-10-18T17:55:47.921ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 475059 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = b4c0fc5b-77e3-4ed4-a6cc-767be1a81ee3 response_code = 500 uri = /omicron-zones
20352024-10-18T17:55:47.921ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b4c0fc5b-77e3-4ed4-a6cc-767be1a81ee3", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b4c0fc5b-77e3-4ed4-a6cc-767be1a81ee3" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 2.691016024s sled_address = [fd00:1122:3344:101::1]:12345
20362024-10-18T17:55:50.621ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
20372024-10-18T17:55:50.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
20382024-10-18T17:55:50.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
20392024-10-18T17:55:50.644ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
20402024-10-18T17:55:50.644ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
20412024-10-18T17:55:50.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
20422024-10-18T17:55:50.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20432024-10-18T17:55:50.682ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
20442024-10-18T17:55:50.685ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
20452024-10-18T17:55:50.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
20462024-10-18T17:55:50.702ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
20472024-10-18T17:55:50.720ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
20482024-10-18T17:55:50.720ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
20492024-10-18T17:55:50.737ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
20502024-10-18T17:55:50.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
20512024-10-18T17:55:50.756ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
20522024-10-18T17:55:50.756ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
20532024-10-18T17:55:50.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
20542024-10-18T17:55:50.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20552024-10-18T17:55:50.792ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
20562024-10-18T17:55:50.792ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20572024-10-18T17:55:50.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
20582024-10-18T17:55:50.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20592024-10-18T17:55:50.827ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20602024-10-18T17:55:50.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
20612024-10-18T17:55:50.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20622024-10-18T17:55:50.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
20632024-10-18T17:55:50.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20642024-10-18T17:55:50.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
20652024-10-18T17:55:50.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20662024-10-18T17:55:50.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
20672024-10-18T17:55:50.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20682024-10-18T17:55:50.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
20692024-10-18T17:55:50.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20702024-10-18T17:55:50.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
20712024-10-18T17:55:50.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20722024-10-18T17:55:50.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
20732024-10-18T17:55:50.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20742024-10-18T17:55:50.958ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
20752024-10-18T17:55:50.973ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
20762024-10-18T17:55:50.973ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
20772024-10-18T17:55:50.991ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20782024-10-18T17:55:50.991ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
20792024-10-18T17:55:50.991ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
20802024-10-18T17:55:50.991ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20812024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
20822024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
20832024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
20842024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
20852024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
20862024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
20872024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
20882024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
20892024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
20902024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
20912024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
20922024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
20932024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
20942024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
20952024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
20962024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
20972024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
20982024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
20992024-10-18T17:55:51.084ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
21002024-10-18T17:55:51.084ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 461300 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 0abe17fe-ae4a-4ec1-95ab-e2f2af523e7a response_code = 500 uri = /omicron-zones
21012024-10-18T17:55:51.084ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0abe17fe-ae4a-4ec1-95ab-e2f2af523e7a", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0abe17fe-ae4a-4ec1-95ab-e2f2af523e7a" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 3.19832715s sled_address = [fd00:1122:3344:101::1]:12345
21022024-10-18T17:55:54.292ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
21032024-10-18T17:55:54.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21042024-10-18T17:55:54.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
21052024-10-18T17:55:54.314ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21062024-10-18T17:55:54.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
21072024-10-18T17:55:54.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21082024-10-18T17:55:54.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21092024-10-18T17:55:54.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21102024-10-18T17:55:54.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
21112024-10-18T17:55:54.372ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21122024-10-18T17:55:54.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
21132024-10-18T17:55:54.390ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21142024-10-18T17:55:54.390ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21152024-10-18T17:55:54.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21162024-10-18T17:55:54.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21172024-10-18T17:55:54.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21182024-10-18T17:55:54.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21192024-10-18T17:55:54.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21202024-10-18T17:55:54.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21212024-10-18T17:55:54.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21222024-10-18T17:55:54.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21232024-10-18T17:55:54.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
21242024-10-18T17:55:54.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21252024-10-18T17:55:54.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21262024-10-18T17:55:54.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
21272024-10-18T17:55:54.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21282024-10-18T17:55:54.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
21292024-10-18T17:55:54.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21302024-10-18T17:55:54.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
21312024-10-18T17:55:54.557ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21322024-10-18T17:55:54.557ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
21332024-10-18T17:55:54.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21342024-10-18T17:55:54.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
21352024-10-18T17:55:54.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21362024-10-18T17:55:54.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
21372024-10-18T17:55:54.612ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21382024-10-18T17:55:54.612ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
21392024-10-18T17:55:54.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21402024-10-18T17:55:54.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
21412024-10-18T17:55:54.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21422024-10-18T17:55:54.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
21432024-10-18T17:55:54.667ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21442024-10-18T17:55:54.667ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
21452024-10-18T17:55:54.667ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
21462024-10-18T17:55:54.668ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21472024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
21482024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
21492024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
21502024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
21512024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
21522024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
21532024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
21542024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
21552024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
21562024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
21572024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
21582024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
21592024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
21602024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
21612024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
21622024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
21632024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
21642024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
21652024-10-18T17:55:54.762ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
21662024-10-18T17:55:54.762ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615", TimeNotSynchronized), ("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd", TimeNotSynchronized), ("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f", TimeNotSynchronized), ("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e", TimeNotSynchronized), ("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63", TimeNotSynchronized), ("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83", TimeNotSynchronized), ("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83", TimeNotSynchronized), ("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b", TimeNotSynchronized), ("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39", TimeNotSynchronized), ("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca", TimeNotSynchronized), ("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e", TimeNotSynchronized), ("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd", TimeNotSynchronized), ("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09", TimeNotSynchronized), ("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49", TimeNotSynchronized), ("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a", TimeNotSynchronized), ("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673", TimeNotSynchronized), ("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330", TimeNotSynchronized), ("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2", TimeNotSynchronized), ("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 468607 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = c4cd1dcd-5d5b-4cfb-b41c-2f775bae57a0 response_code = 500 uri = /omicron-zones
21672024-10-18T17:55:54.763ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c4cd1dcd-5d5b-4cfb-b41c-2f775bae57a0", "content-length": "124", "date": "Fri, 18 Oct 2024 17:55:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c4cd1dcd-5d5b-4cfb-b41c-2f775bae57a0" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 3.826498044s sled_address = [fd00:1122:3344:101::1]:12345
21682024-10-18T17:55:58.600ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 64822dce-d64a-403f-bf53-d84635424320 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 640c31ba-0a33-4d06-bcaa-280a64058d07, kind: Service { id: 64822dce-d64a-403f-bf53-d84635424320 }, name: Name("ntp-64822dce-d64a-403f-bf53-d84635424320"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 131])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 03d0126b-4566-466f-bb9c-de9896115c34 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 69bfa368-1d69-4d23-88e9-3383e573f0eb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68422c74-fc22-434f-add8-652fc427fc03 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2d1c58db-a735-48db-90ac-891756bdde5d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1213c740-3b34-4c05-9e32-d5fae22eaaf8 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c8bd7410-7132-42bd-a55f-2a198a6b2acd (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: c4709288-9135-419c-a897-87bc98b8ce39 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 559ebce8-c652-447b-8465-50ce9559421e (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2a34aa70-afc1-4c49-b577-50951425e4ca (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 8f12f963-ce2b-4e5c-ac07-73f657227673 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External } } } }, OmicronZoneConfig { id: b698f337-7ba4-4ae4-8547-b207e6fcae83 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External } } } }, OmicronZoneConfig { id: 8efe69b7-0efd-43f2-8275-553df5dac16e (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4736466c-52e2-4b18-8ead-0adae93f7d49 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3925fdff-229a-4176-baab-54ab3ed1bb83 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External } } } }, OmicronZoneConfig { id: d6322d41-765c-4dcb-a58b-608a1681fd63 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External } } } }, OmicronZoneConfig { id: e87bc7fc-d008-4cd1-83dd-c64d8728793a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External } } } }, OmicronZoneConfig { id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External } } } }, OmicronZoneConfig { id: a46900f5-d79b-4c04-ad72-7cd4485153fd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
21692024-10-18T17:55:58.601ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21702024-10-18T17:55:58.601ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 5a24845a-5b82-4ab6-b0ce-11d63d68b9cb, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: InternalDns }, responder: ... }
21712024-10-18T17:55:58.623ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21722024-10-18T17:55:58.625ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1fb67d4e-bac4-4e8f-b10a-4ef050bccb88, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: InternalDns }, responder: ... }
21732024-10-18T17:55:58.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21742024-10-18T17:55:58.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7a37cb7b-e967-4955-ad9c-4441ce0d32e6, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21752024-10-18T17:55:58.661ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21762024-10-18T17:55:58.661ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 03d0126b-4566-466f-bb9c-de9896115c34, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Cockroach }, responder: ... }
21772024-10-18T17:55:58.674ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65101
21782024-10-18T17:55:58.674ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 76 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65101 req_id = 9746584c-8ee8-4064-8077-2be9a01e2b5e response_code = 200 uri = /baseboard
21792024-10-18T17:55:58.681ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21802024-10-18T17:55:58.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8e79611e-ba88-4a64-8bf6-28d7df64e66a, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Cockroach }, responder: ... }
21812024-10-18T17:55:58.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21822024-10-18T17:55:58.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 69bfa368-1d69-4d23-88e9-3383e573f0eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21832024-10-18T17:55:58.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21842024-10-18T17:55:58.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68422c74-fc22-434f-add8-652fc427fc03, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21852024-10-18T17:55:58.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
21862024-10-18T17:55:58.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2d1c58db-a735-48db-90ac-891756bdde5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21872024-10-18T17:55:58.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21882024-10-18T17:55:58.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 68483b33-33b2-4e6b-bcd7-72f28a004330, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21892024-10-18T17:55:58.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
21902024-10-18T17:55:58.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21912024-10-18T17:55:58.792ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
21922024-10-18T17:55:58.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c8bd7410-7132-42bd-a55f-2a198a6b2acd, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21932024-10-18T17:55:58.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21942024-10-18T17:55:58.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8f12f963-ce2b-4e5c-ac07-73f657227673, dataset_name: DatasetName { pool_name: ZpoolName { id: 8853a1c1-d102-4f80-9012-356473e594ff (zpool), kind: External }, kind: Crucible }, responder: ... }
21952024-10-18T17:55:58.829ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21962024-10-18T17:55:58.829ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b698f337-7ba4-4ae4-8547-b207e6fcae83, dataset_name: DatasetName { pool_name: ZpoolName { id: 8523f2c5-b045-4f20-8f22-89f361c6e33c (zpool), kind: External }, kind: Crucible }, responder: ... }
21972024-10-18T17:55:58.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
21982024-10-18T17:55:58.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8efe69b7-0efd-43f2-8275-553df5dac16e, dataset_name: DatasetName { pool_name: ZpoolName { id: 24197587-83af-46c0-a36e-5e4a26350d21 (zpool), kind: External }, kind: Crucible }, responder: ... }
21992024-10-18T17:55:58.865ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
22002024-10-18T17:55:58.865ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4736466c-52e2-4b18-8ead-0adae93f7d49, dataset_name: DatasetName { pool_name: ZpoolName { id: 956d8649-5e02-4c83-a766-709d6c1c19f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
22012024-10-18T17:55:58.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
22022024-10-18T17:55:58.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3925fdff-229a-4176-baab-54ab3ed1bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e3d3772-4d1a-4cd4-9451-d3f5402e0376 (zpool), kind: External }, kind: Crucible }, responder: ... }
22032024-10-18T17:55:58.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
22042024-10-18T17:55:58.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d6322d41-765c-4dcb-a58b-608a1681fd63, dataset_name: DatasetName { pool_name: ZpoolName { id: aa3e117f-01a2-4173-8076-71ed9651de49 (zpool), kind: External }, kind: Crucible }, responder: ... }
22052024-10-18T17:55:58.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
22062024-10-18T17:55:58.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e87bc7fc-d008-4cd1-83dd-c64d8728793a, dataset_name: DatasetName { pool_name: ZpoolName { id: e09cb849-5302-41dd-87a5-36e8723c48fb (zpool), kind: External }, kind: Crucible }, responder: ... }
22072024-10-18T17:55:58.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
22082024-10-18T17:55:58.942ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06f6275f-6e08-4df0-8541-1b7a4e8b6615, dataset_name: DatasetName { pool_name: ZpoolName { id: c6223f26-38f9-4483-9d17-efd8e60dd490 (zpool), kind: External }, kind: Crucible }, responder: ... }
22092024-10-18T17:55:58.958ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
22102024-10-18T17:55:58.958ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a46900f5-d79b-4c04-ad72-7cd4485153fd, dataset_name: DatasetName { pool_name: ZpoolName { id: 883d1cbf-e1f8-4c86-b18f-d626feb923fd (zpool), kind: External }, kind: Crucible }, responder: ... }
22112024-10-18T17:55:58.977ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22122024-10-18T17:55:58.978ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
22132024-10-18T17:55:58.978ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
22142024-10-18T17:55:58.978ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22152024-10-18T17:55:59.068ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
22162024-10-18T17:55:59.156ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
22172024-10-18T17:55:59.156ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22182024-10-18T17:55:59.156ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22192024-10-18T17:55:59.156ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22202024-10-18T17:55:59.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22212024-10-18T17:55:59.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22222024-10-18T17:55:59.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22232024-10-18T17:55:59.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22242024-10-18T17:55:59.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22252024-10-18T17:55:59.353ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
22262024-10-18T17:55:59.353ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22272024-10-18T17:55:59.353ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22282024-10-18T17:55:59.353ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22292024-10-18T17:55:59.353ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22302024-10-18T17:55:59.353ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22312024-10-18T17:55:59.354ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
22322024-10-18T17:55:59.366ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
22332024-10-18T17:55:59.377ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22342024-10-18T17:55:59.378ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615 file = illumos-utils/src/zone.rs:316
22352024-10-18T17:55:59.382ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49 file = illumos-utils/src/zone.rs:316
22362024-10-18T17:55:59.386ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63 file = illumos-utils/src/zone.rs:316
22372024-10-18T17:55:59.391ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e file = illumos-utils/src/zone.rs:316
22382024-10-18T17:55:59.395ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd file = illumos-utils/src/zone.rs:316
22392024-10-18T17:55:59.400ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39 file = illumos-utils/src/zone.rs:316
22402024-10-18T17:55:59.405ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd file = illumos-utils/src/zone.rs:316
22412024-10-18T17:55:59.410ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e file = illumos-utils/src/zone.rs:316
22422024-10-18T17:55:59.525ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615 file = illumos-utils/src/zone.rs:354
22432024-10-18T17:55:59.531ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49 file = illumos-utils/src/zone.rs:354
22442024-10-18T17:55:59.535ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63 file = illumos-utils/src/zone.rs:354
22452024-10-18T17:55:59.540ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e file = illumos-utils/src/zone.rs:354
22462024-10-18T17:55:59.544ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd file = illumos-utils/src/zone.rs:354
22472024-10-18T17:55:59.549ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39 file = illumos-utils/src/zone.rs:354
22482024-10-18T17:55:59.554ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd file = illumos-utils/src/zone.rs:354
22492024-10-18T17:55:59.558ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e file = illumos-utils/src/zone.rs:354
22502024-10-18T17:55:59.563ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8 file = illumos-utils/src/zone.rs:316
22512024-10-18T17:55:59.568ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca file = illumos-utils/src/zone.rs:316
22522024-10-18T17:55:59.573ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673 file = illumos-utils/src/zone.rs:316
22532024-10-18T17:55:59.579ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83 file = illumos-utils/src/zone.rs:316
22542024-10-18T17:55:59.585ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83 file = illumos-utils/src/zone.rs:316
22552024-10-18T17:55:59.592ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a file = illumos-utils/src/zone.rs:316
22562024-10-18T17:55:59.597ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8 file = illumos-utils/src/zone.rs:354
22572024-10-18T17:55:59.604ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca file = illumos-utils/src/zone.rs:354
22582024-10-18T17:55:59.609ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673 file = illumos-utils/src/zone.rs:354
22592024-10-18T17:55:59.614ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83 file = illumos-utils/src/zone.rs:354
22602024-10-18T17:55:59.617ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83 file = illumos-utils/src/zone.rs:354
22612024-10-18T17:55:59.623ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a file = illumos-utils/src/zone.rs:354
22622024-10-18T17:55:59.663ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
22632024-10-18T17:55:59.698ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
22642024-10-18T17:55:59.733ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
22652024-10-18T17:55:59.770ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
22662024-10-18T17:55:59.812ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
22672024-10-18T17:55:59.813ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
22682024-10-18T17:55:59.813ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
22692024-10-18T17:55:59.855ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 187, 180]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
22702024-10-18T17:55:59.855ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
22712024-10-18T17:55:59.856ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
22722024-10-18T17:55:59.856ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
22732024-10-18T17:55:59.914ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 222, 198]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
22742024-10-18T17:55:59.914ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
22752024-10-18T17:55:59.915ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
22762024-10-18T17:55:59.915ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
22772024-10-18T17:55:59.976ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 173, 175]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
22782024-10-18T17:55:59.976ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
22792024-10-18T17:55:59.979ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
22802024-10-18T17:55:59.979ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
22812024-10-18T17:56:00.030ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 228, 68]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
22822024-10-18T17:56:00.030ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
22832024-10-18T17:56:00.031ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
22842024-10-18T17:56:00.031ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
22852024-10-18T17:56:00.082ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 185, 152]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
22862024-10-18T17:56:00.082ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
22872024-10-18T17:56:00.082ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
22882024-10-18T17:56:00.082ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
22892024-10-18T17:56:00.083ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
22902024-10-18T17:56:00.083ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
22912024-10-18T17:56:00.083ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
22922024-10-18T17:56:00.083ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
22932024-10-18T17:56:00.083ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
22942024-10-18T17:56:00.086ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22952024-10-18T17:56:00.133ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
22962024-10-18T17:56:00.136ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
22972024-10-18T17:56:00.137ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22982024-10-18T17:56:00.139ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
22992024-10-18T17:56:00.139ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
23002024-10-18T17:56:00.140ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
23012024-10-18T17:56:00.140ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
23022024-10-18T17:56:00.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
23032024-10-18T17:56:00.293ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
23042024-10-18T17:56:00.297ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
23052024-10-18T17:56:00.300ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
23062024-10-18T17:56:00.300ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
23072024-10-18T17:56:00.301ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b file = illumos-utils/src/zone.rs:316
23082024-10-18T17:56:00.315ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 file = illumos-utils/src/zone.rs:316
23092024-10-18T17:56:00.326ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f file = illumos-utils/src/zone.rs:316
23102024-10-18T17:56:00.462ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b file = illumos-utils/src/zone.rs:354
23112024-10-18T17:56:00.475ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 file = illumos-utils/src/zone.rs:354
23122024-10-18T17:56:00.485ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f file = illumos-utils/src/zone.rs:354
23132024-10-18T17:56:00.502ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2 file = illumos-utils/src/zone.rs:316
23142024-10-18T17:56:00.517ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330 file = illumos-utils/src/zone.rs:316
23152024-10-18T17:56:00.544ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2 file = illumos-utils/src/zone.rs:354
23162024-10-18T17:56:00.575ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330 file = illumos-utils/src/zone.rs:354
23172024-10-18T17:56:00.738ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.830849616s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
23182024-10-18T17:56:02.434ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c6223f26-38f9-4483-9d17-efd8e60dd490/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='1ff9245b-2972-419d-ac27-73880f962c55'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23192024-10-18T17:56:02.434ZINFOSledAgent (ServiceManager): Booting oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
23202024-10-18T17:56:02.527ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_24197587-83af-46c0-a36e-5e4a26350d21/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='0f029078-a2f2-48a8-a6b0-b9b3bf0b5295'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23212024-10-18T17:56:02.527ZINFOSledAgent (ServiceManager): Booting oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
23222024-10-18T17:56:02.554ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23232024-10-18T17:56:02.554ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
23242024-10-18T17:56:02.572ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23252024-10-18T17:56:02.573ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
23262024-10-18T17:56:02.650ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_883d1cbf-e1f8-4c86-b18f-d626feb923fd/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='821c0d59-ad53-42c2-a45f-00721d2b7a8a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23272024-10-18T17:56:02.650ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
23282024-10-18T17:56:02.658ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_aa3e117f-01a2-4173-8076-71ed9651de49/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='25938cff-45d3-47cb-968f-bd46a3ceff43'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23292024-10-18T17:56:02.658ZINFOSledAgent (ServiceManager): Booting oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
23302024-10-18T17:56:02.665ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23312024-10-18T17:56:02.665ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
23322024-10-18T17:56:02.673ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_8523f2c5-b045-4f20-8f22-89f361c6e33c/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='392f6388-eaeb-4772-b2b4-4748e57549c6'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23332024-10-18T17:56:02.673ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
23342024-10-18T17:56:02.784ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_956d8649-5e02-4c83-a766-709d6c1c19f4/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3f74f18a-4eb6-49f9-b5fa-a193ab562388'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23352024-10-18T17:56:02.784ZINFOSledAgent (ServiceManager): Booting oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
23362024-10-18T17:56:02.805ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_8853a1c1-d102-4f80-9012-356473e594ff/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5ff38177-a0f1-4560-9abb-ebc164c042c0'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23372024-10-18T17:56:02.805ZINFOSledAgent (ServiceManager): Booting oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
23382024-10-18T17:56:02.869ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e09cb849-5302-41dd-87a5-36e8723c48fb/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='afdd6719-743c-4e79-b6b3-f40a2d8d8eb1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23392024-10-18T17:56:02.869ZINFOSledAgent (ServiceManager): Booting oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
23402024-10-18T17:56:02.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 54.549821ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
23412024-10-18T17:56:03.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 58.662436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
23422024-10-18T17:56:03.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 45.092125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
23432024-10-18T17:56:03.010ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='1213c740-3b34-4c05-9e32-d5fae22eaaf8'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23442024-10-18T17:56:03.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 71.020979ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
23452024-10-18T17:56:03.019ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
23462024-10-18T17:56:03.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 54.706906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
23472024-10-18T17:56:03.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 40.35853ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
23482024-10-18T17:56:03.041ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_4e3d3772-4d1a-4cd4-9451-d3f5402e0376/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='285b6ca2-a27b-4d54-bd1f-900c07b48d3d'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23492024-10-18T17:56:03.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 44.01469ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
23502024-10-18T17:56:03.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 32.664976ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
23512024-10-18T17:56:03.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 53.790346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
23522024-10-18T17:56:03.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 78.875383ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
23532024-10-18T17:56:03.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 51.728454ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
23542024-10-18T17:56:03.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 98.214274ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
23552024-10-18T17:56:03.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 69.922334ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
23562024-10-18T17:56:03.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 105.320773ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
23572024-10-18T17:56:03.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 107.180539ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
23582024-10-18T17:56:03.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 146.384581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
23592024-10-18T17:56:03.662ZINFOSledAgent (ServiceManager): Booting oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
23602024-10-18T17:56:03.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 30.21898ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
23612024-10-18T17:56:03.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 54.514395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
23622024-10-18T17:56:03.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 104.914147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
23632024-10-18T17:56:03.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 146.58859ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
23642024-10-18T17:56:03.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 203.313621ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
23652024-10-18T17:56:04.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 52.004071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
23662024-10-18T17:56:04.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 196.090389ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
23672024-10-18T17:56:04.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 235.995066ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
23682024-10-18T17:56:04.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 26.965202ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
23692024-10-18T17:56:04.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 224.404766ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
23702024-10-18T17:56:04.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 175.391253ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
23712024-10-18T17:56:04.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 183.030928ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
23722024-10-18T17:56:04.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 110.334343ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
23732024-10-18T17:56:04.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 113.703765ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
23742024-10-18T17:56:04.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 290.448607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
23752024-10-18T17:56:04.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 230.006656ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
23762024-10-18T17:56:04.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 332.606301ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
23772024-10-18T17:56:04.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 276.168599ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
23782024-10-18T17:56:04.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 501.344648ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
23792024-10-18T17:56:04.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 426.246184ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
23802024-10-18T17:56:04.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 85.967449ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
23812024-10-18T17:56:04.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 474.436154ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
23822024-10-18T17:56:04.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 491.529807ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
23832024-10-18T17:56:04.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 479.226867ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
23842024-10-18T17:56:04.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 32.312381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
23852024-10-18T17:56:04.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 134.663401ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
23862024-10-18T17:56:04.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 414.499303ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
23872024-10-18T17:56:04.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 295.819656ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
23882024-10-18T17:56:04.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 490.496701ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
23892024-10-18T17:56:04.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 166.795617ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
23902024-10-18T17:56:04.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 53.619532ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
23912024-10-18T17:56:04.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 406.626864ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
23922024-10-18T17:56:04.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 956.104346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
23932024-10-18T17:56:05.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 257.952568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
23942024-10-18T17:56:05.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 218.327871ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
23952024-10-18T17:56:05.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 440.457826ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
23962024-10-18T17:56:05.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 712.921009ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
23972024-10-18T17:56:05.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 775.515933ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
23982024-10-18T17:56:05.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 555.767812ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
23992024-10-18T17:56:05.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 547.662622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24002024-10-18T17:56:05.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 277.54011ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
24012024-10-18T17:56:05.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.063263529s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
24022024-10-18T17:56:05.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 592.6019ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24032024-10-18T17:56:05.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 855.273832ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
24042024-10-18T17:56:05.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 997.311419ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
24052024-10-18T17:56:05.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 1.127656576s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
24062024-10-18T17:56:05.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 981.573531ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
24072024-10-18T17:56:05.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 650.05281ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24082024-10-18T17:56:05.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 1.182337861s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
24092024-10-18T17:56:05.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 1.324972928s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
24102024-10-18T17:56:05.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 1.44627389s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24112024-10-18T17:56:05.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 828.477998ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
24122024-10-18T17:56:05.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 882.192497ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24132024-10-18T17:56:10.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 989.31931ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24142024-10-18T17:56:11.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.014791919s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
24152024-10-18T17:56:11.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 908.249294ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24162024-10-18T17:56:11.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 656.457625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
24172024-10-18T17:56:11.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 1.153607342s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
24182024-10-18T17:56:11.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 1.479910348s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
24192024-10-18T17:56:11.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 1.464553758s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
24202024-10-18T17:56:11.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 1.408655719s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
24212024-10-18T17:56:11.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 877.182248ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24222024-10-18T17:56:11.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 968.534955ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
24232024-10-18T17:56:11.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 855.480949ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
24242024-10-18T17:56:11.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 963.561088ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
24252024-10-18T17:56:11.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 619.486706ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24262024-10-18T17:56:11.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 981.514787ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24272024-10-18T17:56:11.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 1.441745767s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
24282024-10-18T17:56:11.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 1.291119658s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24292024-10-18T17:56:12.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 684.271739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24302024-10-18T17:56:12.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.222894581s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
24312024-10-18T17:56:12.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 1.322810802s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
24322024-10-18T17:56:12.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 799.802053ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24332024-10-18T17:56:12.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 609.597533ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
24342024-10-18T17:56:12.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 1.29278357s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
24352024-10-18T17:56:12.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 1.394419909s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
24362024-10-18T17:56:12.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 875.796788ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
24372024-10-18T17:56:12.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 863.467065ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24382024-10-18T17:56:12.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 996.5849ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
24392024-10-18T17:56:12.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 1.332402801s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
24402024-10-18T17:56:12.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 963.765378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24412024-10-18T17:56:12.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 592.331117ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
24422024-10-18T17:56:12.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 754.718098ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24432024-10-18T17:56:13.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 683.623294ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24442024-10-18T17:56:13.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 676.504687ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
24452024-10-18T17:56:13.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 940.352157ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
24462024-10-18T17:56:13.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 1.2198965s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
24472024-10-18T17:56:13.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 905.183568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
24482024-10-18T17:56:13.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 831.584551ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
24492024-10-18T17:56:13.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 936.055378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24502024-10-18T17:56:13.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 1.255323094s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
24512024-10-18T17:56:13.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 1.242380455s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
24522024-10-18T17:56:13.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 849.540906ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24532024-10-18T17:56:13.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 575.747761ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24542024-10-18T17:56:13.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 634.216343ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
24552024-10-18T17:56:13.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 602.254136ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24562024-10-18T17:56:14.049ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 1.390796794s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
24572024-10-18T17:56:14.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 806.628789ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
24582024-10-18T17:56:14.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.163578556s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
24592024-10-18T17:56:14.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 854.94387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24602024-10-18T17:56:14.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 1.11396737s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
24612024-10-18T17:56:14.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 906.851837ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
24622024-10-18T17:56:14.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 912.04159ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
24632024-10-18T17:56:14.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 520.152239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24642024-10-18T17:56:14.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 1.282708251s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24652024-10-18T17:56:14.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 944.932233ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24662024-10-18T17:56:14.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 852.096992ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
24672024-10-18T17:56:14.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 842.036261ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
24682024-10-18T17:56:14.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 932.955218ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
24692024-10-18T17:56:15.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 1.439315313s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
24702024-10-18T17:56:15.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 1.417269507s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24712024-10-18T17:56:15.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 637.537424ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24722024-10-18T17:56:15.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 1.053654599s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
24732024-10-18T17:56:15.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 1.083861717s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
24742024-10-18T17:56:15.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 1.251367927s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
24752024-10-18T17:56:15.485ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
24762024-10-18T17:56:15.486ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
24772024-10-18T17:56:15.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 628.084092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
24782024-10-18T17:56:15.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 996.154215ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
24792024-10-18T17:56:15.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 555.677892ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24802024-10-18T17:56:15.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 1.426551856s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
24812024-10-18T17:56:15.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 1.35682782s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
24822024-10-18T17:56:15.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 71.188605ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
24832024-10-18T17:56:15.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 537.370083ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
24842024-10-18T17:56:15.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 85.681233ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
24852024-10-18T17:56:15.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 1.168889014s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
24862024-10-18T17:56:15.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 1.398142192s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
24872024-10-18T17:56:16.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 133.611195ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
24882024-10-18T17:56:16.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 376.098368ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
24892024-10-18T17:56:16.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 1.469884694s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
24902024-10-18T17:56:16.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.273202097s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
24912024-10-18T17:56:16.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 765.321543ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
24922024-10-18T17:56:16.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 737.700536ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
24932024-10-18T17:56:16.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 681.086349ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
24942024-10-18T17:56:16.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 1.236119881s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
24952024-10-18T17:56:16.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 908.272589ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
24962024-10-18T17:56:16.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 1.01168372s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
24972024-10-18T17:56:16.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 723.780113ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
24982024-10-18T17:56:16.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 1.127847759s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
24992024-10-18T17:56:16.822ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25002024-10-18T17:56:16.823ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25012024-10-18T17:56:16.823ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25022024-10-18T17:56:16.830ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25032024-10-18T17:56:17.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 1.206120532s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
25042024-10-18T17:56:17.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 690.095848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
25052024-10-18T17:56:17.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 1.173936596s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
25062024-10-18T17:56:17.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 29.33532ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25072024-10-18T17:56:17.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 48.361391ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25082024-10-18T17:56:17.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 680.646397ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
25092024-10-18T17:56:17.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 72.94052ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25102024-10-18T17:56:17.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 1.389658624s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
25112024-10-18T17:56:17.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 838.468823ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
25122024-10-18T17:56:17.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 73.190131ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25132024-10-18T17:56:17.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 644.697235ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
25142024-10-18T17:56:17.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 235.171294ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25152024-10-18T17:56:17.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 278.258609ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25162024-10-18T17:56:17.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 1.401549355s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
25172024-10-18T17:56:17.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 701.897115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
25182024-10-18T17:56:17.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.477992526s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
25192024-10-18T17:56:17.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 1.01199569s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
25202024-10-18T17:56:17.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 504.038208ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
25212024-10-18T17:56:17.768ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.552245766s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
25222024-10-18T17:56:17.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 428.539499ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25232024-10-18T17:56:17.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 1.323132198s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
25242024-10-18T17:56:17.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 423.295429ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25252024-10-18T17:56:17.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 931.945944ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
25262024-10-18T17:56:17.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 783.691905ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
25272024-10-18T17:56:17.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 1.341888637s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
25282024-10-18T17:56:18.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 826.067804ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
25292024-10-18T17:56:18.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 687.715026ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
25302024-10-18T17:56:18.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 615.390594ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
25312024-10-18T17:56:18.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 1.091236781s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25322024-10-18T17:56:18.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 789.718585ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
25332024-10-18T17:56:18.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 775.545743ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25342024-10-18T17:56:18.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 1.498642385s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
25352024-10-18T17:56:18.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 1.09927849s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
25362024-10-18T17:56:18.407ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25372024-10-18T17:56:18.407ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25382024-10-18T17:56:18.408ZINFOSledAgent (ServiceManager): Booting oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
25392024-10-18T17:56:18.416ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25402024-10-18T17:56:18.417ZINFOSledAgent (ServiceManager): Booting oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
25412024-10-18T17:56:18.424ZINFOSledAgent (ServiceManager): Booting oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
25422024-10-18T17:56:18.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 778.458645ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
25432024-10-18T17:56:18.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 1.212309247s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
25442024-10-18T17:56:18.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 59.141342ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
25452024-10-18T17:56:18.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 69.912396ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
25462024-10-18T17:56:18.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 1.07155043s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
25472024-10-18T17:56:18.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 67.135992ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
25482024-10-18T17:56:18.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 75.981095ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
25492024-10-18T17:56:18.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 97.973087ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
25502024-10-18T17:56:18.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 142.135766ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
25512024-10-18T17:56:18.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 1.133976733s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
25522024-10-18T17:56:18.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 632.474689ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
25532024-10-18T17:56:19.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 677.00549ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
25542024-10-18T17:56:19.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 266.579829ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
25552024-10-18T17:56:19.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 63.521747ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
25562024-10-18T17:56:19.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 1.302812395s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25572024-10-18T17:56:19.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 1.396134562s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
25582024-10-18T17:56:19.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 569.444447ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
25592024-10-18T17:56:19.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 595.413355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
25602024-10-18T17:56:19.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 775.377458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
25612024-10-18T17:56:19.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 777.668503ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
25622024-10-18T17:56:19.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 738.367095ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25632024-10-18T17:56:19.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 1.270860846s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
25642024-10-18T17:56:19.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 1.201212874s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
25652024-10-18T17:56:19.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 765.482368ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
25662024-10-18T17:56:19.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 354.396092ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
25672024-10-18T17:56:19.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 100.236025ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
25682024-10-18T17:56:19.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 1.191054381s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
25692024-10-18T17:56:19.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 1.258286131s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
25702024-10-18T17:56:19.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 229.650182ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
25712024-10-18T17:56:19.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 1.391702687s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
25722024-10-18T17:56:19.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 935.514562ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
25732024-10-18T17:56:19.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 1.488197048s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
25742024-10-18T17:56:19.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 654.53862ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
25752024-10-18T17:56:19.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 652.910545ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
25762024-10-18T17:56:19.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 1.125402398s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
25772024-10-18T17:56:19.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 728.474459ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
25782024-10-18T17:56:19.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.287574542s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
25792024-10-18T17:56:20.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 503.392407ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
25802024-10-18T17:56:20.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 1.187164577s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
25812024-10-18T17:56:20.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 982.639006ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
25822024-10-18T17:56:20.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 745.036146ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
25832024-10-18T17:56:20.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83") failed: Property not found. retry in 1.260470163s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
25842024-10-18T17:56:20.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 1.134175142s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
25852024-10-18T17:56:20.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 873.314593ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
25862024-10-18T17:56:20.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 1.328536604s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
25872024-10-18T17:56:20.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd") failed: Property not found. retry in 1.439697268s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
25882024-10-18T17:56:20.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 1.449637828s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
25892024-10-18T17:56:20.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a") failed: Property not found. retry in 877.436397ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
25902024-10-18T17:56:20.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca") failed: Property not found. retry in 1.029750605s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
25912024-10-18T17:56:20.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83") failed: Property not found. retry in 1.221966967s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
25922024-10-18T17:56:20.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e") failed: Property not found. retry in 940.312274ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
25932024-10-18T17:56:20.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 1.379308498s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
25942024-10-18T17:56:20.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63") failed: Property not found. retry in 946.889249ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
25952024-10-18T17:56:20.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e") failed: Property not found. retry in 533.545462ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
25962024-10-18T17:56:21.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39") failed: Property not found. retry in 739.35047ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
25972024-10-18T17:56:21.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8") failed: Property not found. retry in 1.054632198s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
25982024-10-18T17:56:21.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49") failed: Property not found. retry in 741.464044ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
25992024-10-18T17:56:21.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615") failed: Property not found. retry in 1.209058182s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
26002024-10-18T17:56:21.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 1.108388966s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
26012024-10-18T17:56:21.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673") failed: Property not found. retry in 1.24029731s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
26022024-10-18T17:56:21.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 1.362454223s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
26032024-10-18T17:56:21.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 1.494619558s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
26042024-10-18T17:56:22.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 768.020972ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
26052024-10-18T17:56:22.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 645.154917ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
26062024-10-18T17:56:22.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 668.694254ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
26072024-10-18T17:56:22.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 1.366620589s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
26082024-10-18T17:56:22.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 818.725372ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
26092024-10-18T17:56:22.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 536.502235ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
26102024-10-18T17:56:23.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 1.051966836s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
26112024-10-18T17:56:23.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 1.424437543s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
26122024-10-18T17:56:23.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 1.389105989s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
26132024-10-18T17:56:23.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 921.047716ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
26142024-10-18T17:56:23.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 563.232735ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
26152024-10-18T17:56:23.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 571.390468ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
26162024-10-18T17:56:24.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 959.253747ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
26172024-10-18T17:56:24.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 1.224722709s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
26182024-10-18T17:56:24.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 630.956067ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
26192024-10-18T17:56:24.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 892.907522ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
26202024-10-18T17:56:24.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 1.01970516s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
26212024-10-18T17:56:24.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 1.237427967s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
26222024-10-18T17:56:25.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 1.14202864s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
26232024-10-18T17:56:25.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 877.547397ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
26242024-10-18T17:56:25.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b") failed: Property not found. retry in 1.367653821s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
26252024-10-18T17:56:25.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd") failed: Property not found. retry in 1.212097347s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
26262024-10-18T17:56:25.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 649.573113ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
26272024-10-18T17:56:26.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f") failed: Property not found. retry in 1.190194426s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
26282024-10-18T17:56:26.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330") failed: Property not found. retry in 634.214518ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
26292024-10-18T17:56:26.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09") failed: Property not found. retry in 941.807834ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
26302024-10-18T17:56:26.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2") failed: Property not found. retry in 849.912338ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
26312024-10-18T17:56:27.072ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.203115989s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
26322024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_30ebfc02-3884-4b1b-b0ab-82f344558c6b
26332024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_06f6275f-6e08-4df0-8541-1b7a4e8b6615
26342024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_4736466c-52e2-4b18-8ead-0adae93f7d49
26352024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_d6322d41-765c-4dcb-a58b-608a1681fd63
26362024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_8efe69b7-0efd-43f2-8275-553df5dac16e
26372024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_clickhouse_c8bd7410-7132-42bd-a55f-2a198a6b2acd
26382024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_c4709288-9135-419c-a897-87bc98b8ce39
26392024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_a46900f5-d79b-4c04-ad72-7cd4485153fd
26402024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_559ebce8-c652-447b-8465-50ce9559421e
26412024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_external_dns_28a703d2-5c3e-4a89-8c59-9a6b18fe5d09
26422024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_oximeter_1213c740-3b34-4c05-9e32-d5fae22eaaf8
26432024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_2a34aa70-afc1-4c49-b577-50951425e4ca
26442024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_8f12f963-ce2b-4e5c-ac07-73f657227673
26452024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_b698f337-7ba4-4ae4-8547-b207e6fcae83
26462024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_3925fdff-229a-4176-baab-54ab3ed1bb83
26472024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_c5f97bb5-dce0-4358-bd22-da6cb1e4812f
26482024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_c3438018-9a74-4aff-bf04-5f7fce86d0b2
26492024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_external_dns_68483b33-33b2-4e6b-bcd7-72f28a004330
26502024-10-18T17:56:27.237ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_e87bc7fc-d008-4cd1-83dd-c64d8728793a
26512024-10-18T17:56:27.238ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
26522024-10-18T17:56:27.240ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
26532024-10-18T17:56:27.246ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 28601152 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34297 req_id = 57724a7d-df2a-42ab-95db-7bea6f34fb30 response_code = 204 uri = /omicron-zones
26542024-10-18T17:56:27.255ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1321
26552024-10-18T17:56:27.256ZINFOSledAgent (RSS): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
26562024-10-18T17:56:27.257ZINFOSledAgent (RSS): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
26572024-10-18T17:56:27.264ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
26582024-10-18T17:56:27.266ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::a]:12221 file = sled-agent/src/rack_setup/service.rs:719
26592024-10-18T17:56:27.318ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, rib_priority: None, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:886
26602024-10-18T17:56:27.319ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26612024-10-18T17:56:27.389ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26622024-10-18T17:56:27.563ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26632024-10-18T17:56:27.728ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26642024-10-18T17:56:27.962ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26652024-10-18T17:56:28.208ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26662024-10-18T17:56:28.580ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26672024-10-18T17:56:28.789ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26682024-10-18T17:56:28.824ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59095
26692024-10-18T17:56:28.828ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1273 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59095 req_id = fe0efb77-a0b9-4cdc-b3de-ab757d20b4bc response_code = 200 uri = /baseboard
26702024-10-18T17:56:29.044ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26712024-10-18T17:56:29.501ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26722024-10-18T17:56:30.194ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26732024-10-18T17:56:31.004ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26742024-10-18T17:56:31.312ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.002000096s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
26752024-10-18T17:56:31.461ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26762024-10-18T17:56:31.908ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26772024-10-18T17:56:32.569ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26782024-10-18T17:56:33.407ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26792024-10-18T17:56:35.503ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/33bc13e3-6beb-4d1b-8b04-6e3da281e5d7/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
26802024-10-18T17:56:38.318ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "fdb30928-805b-47ba-8ff3-3da6f3f58ad7", "content-length": "173", "date": "Fri, 18 Oct 2024 17:56:38 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"33bc13e3-6beb-4d1b-8b04-6e3da281e5d7\"", request_id: "fdb30928-805b-47ba-8ff3-3da6f3f58ad7" } file = sled-agent/src/rack_setup/service.rs:948
26812024-10-18T17:56:41.623ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "c4c4862b-d825-4e26-91a5-efd02fb9ff41", "content-length": "173", "date": "Fri, 18 Oct 2024 17:56:41 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"08066c3d-b7ed-49d2-8578-f8769fb94736\"", request_id: "c4c4862b-d825-4e26-91a5-efd02fb9ff41" } file = sled-agent/src/rack_setup/service.rs:948
26822024-10-18T17:56:43.625ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "a72bda82-1ffb-4404-b39a-eb15e4ecdeef", "content-length": "173", "date": "Fri, 18 Oct 2024 17:56:43 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"08066c3d-b7ed-49d2-8578-f8769fb94736\"", request_id: "a72bda82-1ffb-4404-b39a-eb15e4ecdeef" } file = sled-agent/src/rack_setup/service.rs:948
26832024-10-18T17:56:48.671ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "6a0accce-cdac-45d5-b471-4e4442331f6b", "content-length": "173", "date": "Fri, 18 Oct 2024 17:56:48 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"08066c3d-b7ed-49d2-8578-f8769fb94736\"", request_id: "6a0accce-cdac-45d5-b471-4e4442331f6b" } file = sled-agent/src/rack_setup/service.rs:948
26842024-10-18T17:56:48.972ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:500 new_generation = 1 old_generation = 1 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
26852024-10-18T17:56:49.057ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33841
26862024-10-18T17:56:49.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
26872024-10-18T17:56:49.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
26882024-10-18T17:56:49.059ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
26892024-10-18T17:56:49.060ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
26902024-10-18T17:56:49.251ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 192741 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33841 req_id = 45205433-5d97-4555-8d06-a098588ad2c5 response_code = 200 uri = /inventory
26912024-10-18T17:56:54.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52914
26922024-10-18T17:56:54.301ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26932024-10-18T17:56:54.302ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26942024-10-18T17:56:54.303ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26952024-10-18T17:56:54.303ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
26962024-10-18T17:56:54.303ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26972024-10-18T17:56:54.303ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26982024-10-18T17:56:54.303ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26992024-10-18T17:56:54.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3779 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52914 req_id = 57ba22df-059e-4ef0-b533-c1de7d534e29 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27002024-10-18T17:56:54.304ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:958
27012024-10-18T17:56:54.489ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59030
27022024-10-18T17:56:54.490ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
27032024-10-18T17:56:54.490ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
27042024-10-18T17:56:54.490ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/config/omicron-zones.json file = common/src/ledger.rs:177
27052024-10-18T17:56:54.490ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/config/omicron-zones.json file = common/src/ledger.rs:177
27062024-10-18T17:56:54.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 180519 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59030 req_id = a0f0d445-343f-4b2e-b971-d4b8a7068e28 response_code = 200 uri = /inventory
27072024-10-18T17:56:55.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55381
27082024-10-18T17:56:55.139ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
27092024-10-18T17:56:55.139ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27102024-10-18T17:56:55.139ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27112024-10-18T17:56:55.139ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
27122024-10-18T17:56:55.139ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27132024-10-18T17:56:55.139ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27142024-10-18T17:56:55.139ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27152024-10-18T17:56:55.139ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 330 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55381 req_id = ddc2ea25-fbe4-4d99-b850-1eebb528702d response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27162024-10-18T17:56:55.427ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56298
27172024-10-18T17:56:55.427ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
27182024-10-18T17:56:55.427ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27192024-10-18T17:56:55.427ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27202024-10-18T17:56:55.427ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
27212024-10-18T17:56:55.427ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27222024-10-18T17:56:55.427ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27232024-10-18T17:56:55.427ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27242024-10-18T17:56:55.427ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 282 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56298 req_id = 5f205570-d0c1-4499-8dde-ce83296d29ab response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27252024-10-18T17:56:55.533ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61960
27262024-10-18T17:56:55.533ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 36ba59af-bf85-4423-a357-51f0d41b09ff, kind: Service { id: 68483b33-33b2-4e6b-bcd7-72f28a004330 }, name: Name("external-dns-68483b33-33b2-4e6b-bcd7-72f28a004330"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 152])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f1929d9e-6f3d-4be3-b8e4-711b38867554, kind: Service { id: 28a703d2-5c3e-4a89-8c59-9a6b18fe5d09 }, name: Name("external-dns-28a703d2-5c3e-4a89-8c59-9a6b18fe5d09"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 198])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 53cfda59-d0cc-456d-8218-38fa278d1ca8, kind: Service { id: 30ebfc02-3884-4b1b-b0ab-82f344558c6b }, name: Name("nexus-30ebfc02-3884-4b1b-b0ab-82f344558c6b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8be92f42-3bcf-4463-9be3-fc3eaa3e55bb, kind: Service { id: c3438018-9a74-4aff-bf04-5f7fce86d0b2 }, name: Name("nexus-c3438018-9a74-4aff-bf04-5f7fce86d0b2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 68])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f67c0489-2953-40c9-ae7c-0422d543c65f, kind: Service { id: c5f97bb5-dce0-4358-bd22-da6cb1e4812f }, name: Name("nexus-c5f97bb5-dce0-4358-bd22-da6cb1e4812f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 175])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
27272024-10-18T17:56:55.533ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27282024-10-18T17:56:55.533ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27292024-10-18T17:56:55.534ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
27302024-10-18T17:56:55.534ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27312024-10-18T17:56:55.534ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27322024-10-18T17:56:55.534ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27332024-10-18T17:56:55.534ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 298 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61960 req_id = 1ff159d2-eff4-45ed-9d13-5b7150c7fa41 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27342024-10-18T17:56:58.869ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52047
27352024-10-18T17:56:58.869ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52047 req_id = cfeb2b28-858f-42b2-8177-c5db15b52da3 response_code = 200 uri = /baseboard
27362024-10-18T17:57:00.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61022
27372024-10-18T17:57:00.940ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61022 req_id = 61bc5ee6-47f1-4e5c-ba7d-74c72a51e3cd response_code = 200 uri = /vpc-routes
27382024-10-18T17:57:00.982ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
27392024-10-18T17:57:00.982ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61022 req_id = 362a0b46-b58b-4994-b16d-59d51de818f4 response_code = 204 uri = /eip-gateways
27402024-10-18T17:57:01.077ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2155 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61022 req_id = 660d326b-e632-45ca-adcc-d17c79811109 response_code = 204 uri = /vpc-routes
27412024-10-18T17:57:08.765ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42452
27422024-10-18T17:57:08.766ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 324 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42452 req_id = 043506ec-df89-4b90-af21-2c8fb704cfc8 response_code = 200 uri = /v2p
27432024-10-18T17:57:08.832ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41607
27442024-10-18T17:57:08.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41607 req_id = 10f69ba2-41e8-470a-a1f8-1156fa199ef1 response_code = 200 uri = /vpc-routes
27452024-10-18T17:57:08.900ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
27462024-10-18T17:57:08.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41607 req_id = 2d1457d5-a634-49d5-9c38-a8c03a41df80 response_code = 204 uri = /eip-gateways
27472024-10-18T17:57:08.911ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 678 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41607 req_id = 3922e2ae-1cb4-40e0-8576-110ee15c25b5 response_code = 204 uri = /vpc-routes
27482024-10-18T17:57:08.973ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37640
27492024-10-18T17:57:08.973ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
27502024-10-18T17:57:09.035ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
27512024-10-18T17:57:09.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145686 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:37640 req_id = 96f81eb7-5e15-4202-a275-196902e4f8e8 response_code = 204 uri = /switch-ports
27522024-10-18T17:57:09.120ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 521 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37640 req_id = 06751427-6ed0-4a4c-998e-2f41937ab8c7 response_code = 200 uri = /network-bootstore-config
27532024-10-18T17:57:09.216ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
27542024-10-18T17:57:09.216ZINFOSledAgent (bootstore): Writing ledger to /pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
27552024-10-18T17:57:09.218ZINFOSledAgent (bootstore): Writing ledger to /pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
27562024-10-18T17:57:09.223ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
27572024-10-18T17:57:09.223ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8554 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37640 req_id = 4ed0a662-935e-4ff3-9d8a-02753715b3ee response_code = 204 uri = /network-bootstore-config
27582024-10-18T17:57:09.264ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55945
27592024-10-18T17:57:09.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55945 req_id = e663f0f5-646a-4d9c-af45-6c530c8b88dd response_code = 200 uri = /vpc-routes
27602024-10-18T17:57:09.313ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
27612024-10-18T17:57:09.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55945 req_id = ab996ead-163e-4a81-a154-070f3a081c2c response_code = 204 uri = /eip-gateways
27622024-10-18T17:57:09.328ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55945 req_id = c13b9f44-e1e3-410d-9c9d-cccb0ca26c5a response_code = 204 uri = /vpc-routes
27632024-10-18T17:57:09.332ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36200
27642024-10-18T17:57:09.332ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36200 req_id = 4dc1fd33-5aac-414f-8f59-d23450458ba7 response_code = 200 uri = /v2p
27652024-10-18T17:57:09.413ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58391
27662024-10-18T17:57:09.413ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
27672024-10-18T17:57:09.449ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
27682024-10-18T17:57:09.515ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 101905 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58391 req_id = 87d82781-d7d0-4657-906e-2a4ad28818e9 response_code = 204 uri = /switch-ports
27692024-10-18T17:57:09.515ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58391 req_id = e6f077f3-aa44-44b7-8482-40542ca5076e response_code = 200 uri = /network-bootstore-config
27702024-10-18T17:57:09.606ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52669
27712024-10-18T17:57:09.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52669 req_id = 960e9dae-92a5-4ec6-99f1-8f24592bcdf2 response_code = 200 uri = /vpc-routes
27722024-10-18T17:57:09.654ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
27732024-10-18T17:57:09.654ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52669 req_id = 69021c09-885b-46f7-b4ec-4155d11f7e47 response_code = 204 uri = /eip-gateways
27742024-10-18T17:57:09.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 405 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52669 req_id = 1dc02d86-fa46-46c6-99df-562651b416da response_code = 204 uri = /vpc-routes
27752024-10-18T17:57:09.792ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37565
27762024-10-18T17:57:09.792ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37565 req_id = 1d3d02a8-5aa3-43ec-acfc-562ffc3e72a9 response_code = 200 uri = /v2p
27772024-10-18T17:57:09.995ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42035
27782024-10-18T17:57:09.996ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
27792024-10-18T17:57:10.032ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
27802024-10-18T17:57:10.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92428 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42035 req_id = 517782c2-c66b-4b0c-82e4-9277601450c8 response_code = 204 uri = /switch-ports
27812024-10-18T17:57:10.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42035 req_id = a016d6d0-041e-45ac-9221-ee35e9bf4512 response_code = 200 uri = /network-bootstore-config
27822024-10-18T17:57:10.442ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:65402 remote_addr = [fd00:1122:3344:101::d]:37310 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
27832024-10-18T17:57:10.684ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 241347 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = 8e6f3da3-3b5d-4ef6-8676-3af68fc909d6 response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
27842024-10-18T17:57:28.904ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57868
27852024-10-18T17:57:28.904ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57868 req_id = 80acaca2-da35-4d41-b7fc-2a3dbb4c7b1e response_code = 200 uri = /baseboard
27862024-10-18T17:57:38.770ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61100
27872024-10-18T17:57:38.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61100 req_id = 3a801783-ed08-49d6-bdbb-63c1f1bdb0fe response_code = 200 uri = /vpc-routes
27882024-10-18T17:57:38.816ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
27892024-10-18T17:57:38.816ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61100 req_id = 0a13a834-f3aa-4b54-810d-007aaa1f8d26 response_code = 204 uri = /eip-gateways
27902024-10-18T17:57:38.829ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 272 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61100 req_id = 7112545c-29a8-4170-b39f-71c53654e8b2 response_code = 204 uri = /vpc-routes
27912024-10-18T17:57:38.977ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53322
27922024-10-18T17:57:38.978ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53322 req_id = 530f7ea4-74ef-4f60-9a39-567ae78efd1e response_code = 200 uri = /v2p
27932024-10-18T17:57:39.132ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48469
27942024-10-18T17:57:39.132ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
27952024-10-18T17:57:39.180ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
27962024-10-18T17:57:39.249ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116370 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48469 req_id = c7a9f7a3-1889-45c0-8628-db9dd7983939 response_code = 204 uri = /switch-ports
27972024-10-18T17:57:39.249ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48469 req_id = 0d9a2e75-02aa-445b-b979-2f881ea9eaf8 response_code = 200 uri = /network-bootstore-config
27982024-10-18T17:57:39.369ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44718
27992024-10-18T17:57:39.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44718 req_id = 7a022a7c-9813-4b06-811d-41cc99ec1941 response_code = 200 uri = /vpc-routes
28002024-10-18T17:57:39.385ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28012024-10-18T17:57:39.385ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44718 req_id = 0d0058e7-0c15-4ba5-aafc-00763e6363f3 response_code = 204 uri = /eip-gateways
28022024-10-18T17:57:39.396ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 323 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44718 req_id = 9cf85a39-f452-456f-9bd8-55f2400e3ee1 response_code = 204 uri = /vpc-routes
28032024-10-18T17:57:39.616ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40600
28042024-10-18T17:57:39.616ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 163 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40600 req_id = d3c3082a-ad53-448d-848a-80f084f6632d response_code = 200 uri = /v2p
28052024-10-18T17:57:39.636ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37158
28062024-10-18T17:57:39.636ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28072024-10-18T17:57:39.674ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28082024-10-18T17:57:39.724ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 87437 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37158 req_id = 07299cde-ccd5-4cfa-b11b-bdcbd700cdda response_code = 204 uri = /switch-ports
28092024-10-18T17:57:39.724ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37158 req_id = be8b6b0a-4fe7-4349-95d2-e5745259724d response_code = 200 uri = /network-bootstore-config
28102024-10-18T17:57:39.757ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39420
28112024-10-18T17:57:39.757ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39420 req_id = 75c6c1d9-057b-408b-a237-623fc71582ab response_code = 200 uri = /vpc-routes
28122024-10-18T17:57:39.779ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28132024-10-18T17:57:39.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:39420 req_id = 2e4cf77c-622e-48a0-97e8-145bab410b15 response_code = 204 uri = /eip-gateways
28142024-10-18T17:57:39.884ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43392
28152024-10-18T17:57:39.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 195 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43392 req_id = 8e9585b1-3f0a-49c7-9bf6-ebeae6b3ccc4 response_code = 200 uri = /v2p
28162024-10-18T17:57:39.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:39420 req_id = 49a4de59-9ade-415d-8350-3c1894ae936c response_code = 204 uri = /vpc-routes
28172024-10-18T17:57:39.991ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41969
28182024-10-18T17:57:39.991ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28192024-10-18T17:57:40.029ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28202024-10-18T17:57:40.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97759 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:41969 req_id = 63a6ebf0-97ab-48ab-8fcf-25a58c4dc0c9 response_code = 204 uri = /switch-ports
28212024-10-18T17:57:40.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41969 req_id = 67dd4125-30ca-445a-ab2e-0feb0b2e0a76 response_code = 200 uri = /network-bootstore-config
28222024-10-18T17:57:40.437ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7778 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = b911036a-4ff1-4bbb-a643-f4c6f2b49d6d response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
28232024-10-18T17:57:58.952ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61993
28242024-10-18T17:57:58.952ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 69 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61993 req_id = f5290105-74af-4e53-8fec-240f97c22217 response_code = 200 uri = /baseboard
28252024-10-18T17:58:08.625ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54594
28262024-10-18T17:58:08.625ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54594 req_id = 9bcdac0b-1287-4a7b-9726-e7f6d4592271 response_code = 200 uri = /vpc-routes
28272024-10-18T17:58:08.679ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28282024-10-18T17:58:08.679ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54594 req_id = 988c9e8b-227b-4c0e-b6a9-cc01849d5850 response_code = 204 uri = /eip-gateways
28292024-10-18T17:58:08.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 197 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54594 req_id = b95427ed-760f-4e01-a585-94baedf770bb response_code = 204 uri = /vpc-routes
28302024-10-18T17:58:08.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58781
28312024-10-18T17:58:08.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58781 req_id = d09f3260-4608-458a-b169-4ba9c3e2cd72 response_code = 200 uri = /v2p
28322024-10-18T17:58:08.770ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61148
28332024-10-18T17:58:08.770ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28342024-10-18T17:58:08.806ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28352024-10-18T17:58:08.862ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92388 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:61148 req_id = a1f6feac-5d6e-4bbc-9c14-fc8df6110cca response_code = 204 uri = /switch-ports
28362024-10-18T17:58:08.862ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61148 req_id = c8968df1-a32a-4059-b617-208b0c942b07 response_code = 200 uri = /network-bootstore-config
28372024-10-18T17:58:09.167ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46320
28382024-10-18T17:58:09.167ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46320 req_id = cf31942c-f40c-46e8-bdae-b933371a4f26 response_code = 200 uri = /vpc-routes
28392024-10-18T17:58:09.182ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28402024-10-18T17:58:09.182ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46320 req_id = 53472681-b27b-4ac0-8805-cd0c9fdf1912 response_code = 204 uri = /eip-gateways
28412024-10-18T17:58:09.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 216 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46320 req_id = 9e506788-f528-46e8-ad2f-6c1623f0021c response_code = 204 uri = /vpc-routes
28422024-10-18T17:58:09.281ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36772
28432024-10-18T17:58:09.281ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36772 req_id = de8c0e2a-dd83-47af-ab80-77e39ea84eeb response_code = 200 uri = /v2p
28442024-10-18T17:58:09.363ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59948
28452024-10-18T17:58:09.363ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28462024-10-18T17:58:09.400ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28472024-10-18T17:58:09.454ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90890 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59948 req_id = 0eff6155-0487-475c-928a-ae5bbdf188aa response_code = 204 uri = /switch-ports
28482024-10-18T17:58:09.455ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59948 req_id = 5afe47cc-81a7-461c-98ae-39aa252c0197 response_code = 200 uri = /network-bootstore-config
28492024-10-18T17:58:09.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56049
28502024-10-18T17:58:09.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56049 req_id = 3f47c4e4-3166-494c-aab0-838420d2571a response_code = 200 uri = /vpc-routes
28512024-10-18T17:58:09.734ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28522024-10-18T17:58:09.734ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56049 req_id = 31a807a4-f1bd-41c8-be62-008f8e97301e response_code = 204 uri = /eip-gateways
28532024-10-18T17:58:09.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56049 req_id = cdbc593c-cdd6-4449-a9ec-6f6cba563723 response_code = 204 uri = /vpc-routes
28542024-10-18T17:58:09.827ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56167
28552024-10-18T17:58:09.827ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56167 req_id = f524bcb1-4b25-42da-9683-be94be361187 response_code = 200 uri = /v2p
28562024-10-18T17:58:09.879ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42491
28572024-10-18T17:58:09.879ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28582024-10-18T17:58:09.917ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28592024-10-18T17:58:09.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93267 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42491 req_id = c9580522-e4e0-4110-87a9-764300e50662 response_code = 204 uri = /switch-ports
28602024-10-18T17:58:09.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42491 req_id = 17f57c0a-0474-4605-b9d0-e66c59e7c7ff response_code = 200 uri = /network-bootstore-config
28612024-10-18T17:58:10.436ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7482 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = e53ffb4f-62f9-4da1-96f0-d20e9bcd1a61 response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
28622024-10-18T17:58:28.988ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44476
28632024-10-18T17:58:28.988ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44476 req_id = 8eb7814a-84de-4832-85e3-cb2721ed14f0 response_code = 200 uri = /baseboard
28642024-10-18T17:58:38.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45664
28652024-10-18T17:58:38.690ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45664 req_id = 3f766915-4768-4e36-a842-e5da8fee2778 response_code = 200 uri = /vpc-routes
28662024-10-18T17:58:38.717ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28672024-10-18T17:58:38.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45664 req_id = 8b7e8e8b-9e8d-4b79-9542-d02933994697 response_code = 204 uri = /eip-gateways
28682024-10-18T17:58:38.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 208 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45664 req_id = 46853ca8-76ff-4bf4-b687-0c19747e00e6 response_code = 204 uri = /vpc-routes
28692024-10-18T17:58:39.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36312
28702024-10-18T17:58:39.047ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36312 req_id = d348f1a2-b259-4e2b-bdfa-6c79b4c32d07 response_code = 200 uri = /v2p
28712024-10-18T17:58:39.270ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54003
28722024-10-18T17:58:39.271ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28732024-10-18T17:58:39.314ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28742024-10-18T17:58:39.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103267 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54003 req_id = 74cfee97-53ca-494a-901b-17b1c38893bd response_code = 204 uri = /switch-ports
28752024-10-18T17:58:39.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54003 req_id = 26c98625-9599-4e80-a2e2-4390eb98444e response_code = 200 uri = /network-bootstore-config
28762024-10-18T17:58:39.454ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49394
28772024-10-18T17:58:39.454ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49394 req_id = e450e413-1af7-4387-905e-91dc35a07127 response_code = 200 uri = /vpc-routes
28782024-10-18T17:58:39.627ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28792024-10-18T17:58:39.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49394 req_id = 2a269e00-3a0c-40ee-bc80-c9c0279d3737 response_code = 204 uri = /eip-gateways
28802024-10-18T17:58:39.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49394 req_id = 5e7c8567-930f-4559-ba19-94756fab02a8 response_code = 204 uri = /vpc-routes
28812024-10-18T17:58:39.686ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32901
28822024-10-18T17:58:39.686ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28832024-10-18T17:58:39.732ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28842024-10-18T17:58:39.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109220 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:32901 req_id = f79e27d7-4c2f-46a8-952e-0317b5bc59b8 response_code = 204 uri = /switch-ports
28852024-10-18T17:58:39.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58587
28862024-10-18T17:58:39.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58587 req_id = d56bccbf-c898-4939-a831-d973ffacb8b0 response_code = 200 uri = /v2p
28872024-10-18T17:58:39.796ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32901 req_id = c79ff0e2-758e-4a75-9d54-4366b33847c8 response_code = 200 uri = /network-bootstore-config
28882024-10-18T17:58:39.852ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43508
28892024-10-18T17:58:39.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43508 req_id = b5ab75c0-1f28-410c-9be5-cb91ca34303f response_code = 200 uri = /vpc-routes
28902024-10-18T17:58:39.867ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
28912024-10-18T17:58:39.868ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43508 req_id = c000a7c0-f69d-4aa3-932c-890183be88a7 response_code = 204 uri = /eip-gateways
28922024-10-18T17:58:39.906ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62493
28932024-10-18T17:58:39.907ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62493 req_id = fc569d25-63ca-4ce3-82ab-6a9ed66cc51b response_code = 200 uri = /v2p
28942024-10-18T17:58:39.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 189 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43508 req_id = b111ffbc-82a7-4bb5-b40f-01278ace8040 response_code = 204 uri = /vpc-routes
28952024-10-18T17:58:40.177ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52880
28962024-10-18T17:58:40.177ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
28972024-10-18T17:58:40.215ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
28982024-10-18T17:58:40.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94176 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52880 req_id = 6063c33f-e405-4752-8c73-7b07bd2a52ee response_code = 204 uri = /switch-ports
28992024-10-18T17:58:40.272ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52880 req_id = fd179747-bd29-475e-8ec5-dae720398280 response_code = 200 uri = /network-bootstore-config
29002024-10-18T17:58:40.436ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7073 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = 0441393c-1e42-4d5e-979f-682be1f647fe response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
29012024-10-18T17:58:59.022ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46695
29022024-10-18T17:58:59.022ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 78 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46695 req_id = 692eec89-b6e7-4ff8-90ac-abb4811efe3e response_code = 200 uri = /baseboard
29032024-10-18T17:59:08.681ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49410
29042024-10-18T17:59:08.681ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49410 req_id = 8831e56f-0952-4fc5-a062-59d1bb83508a response_code = 200 uri = /vpc-routes
29052024-10-18T17:59:08.727ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29062024-10-18T17:59:08.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49410 req_id = 06799463-eb57-41c0-9a28-29ab91fbad9f response_code = 204 uri = /eip-gateways
29072024-10-18T17:59:08.871ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54604
29082024-10-18T17:59:08.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54604 req_id = 8ae05879-e2c5-4b7c-aedb-5c8a5050f85c response_code = 200 uri = /v2p
29092024-10-18T17:59:08.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 276 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49410 req_id = 7bbc5487-a73a-4c02-868e-abd321d6b12e response_code = 204 uri = /vpc-routes
29102024-10-18T17:59:08.924ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55172
29112024-10-18T17:59:08.924ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
29122024-10-18T17:59:08.961ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
29132024-10-18T17:59:09.018ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94499 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:55172 req_id = 445582d3-4eb0-45d2-b9bc-3fd5272572b8 response_code = 204 uri = /switch-ports
29142024-10-18T17:59:09.019ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55172 req_id = 19304db9-e48c-49e7-ac09-868bb8875825 response_code = 200 uri = /network-bootstore-config
29152024-10-18T17:59:09.265ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36103
29162024-10-18T17:59:09.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36103 req_id = 330a4a88-69db-4b55-9f7f-6241257cd64a response_code = 200 uri = /vpc-routes
29172024-10-18T17:59:09.304ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29182024-10-18T17:59:09.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36103 req_id = 3c1abfda-e9d7-4d4a-bf0d-101eea899e89 response_code = 204 uri = /eip-gateways
29192024-10-18T17:59:09.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 250 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36103 req_id = ce9eb802-1098-454c-be0e-ec8401fb7016 response_code = 204 uri = /vpc-routes
29202024-10-18T17:59:09.370ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46262
29212024-10-18T17:59:09.371ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46262 req_id = cf154175-7396-437a-8c3c-334ff97195c7 response_code = 200 uri = /v2p
29222024-10-18T17:59:09.383ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44373
29232024-10-18T17:59:09.383ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
29242024-10-18T17:59:09.421ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
29252024-10-18T17:59:09.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96731 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44373 req_id = c0feb7e9-8f6d-49e8-8c7f-715d1e3fc6e8 response_code = 204 uri = /switch-ports
29262024-10-18T17:59:09.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44373 req_id = c5272cc7-1dea-4095-92df-96309cf2e8bf response_code = 200 uri = /network-bootstore-config
29272024-10-18T17:59:09.602ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52452
29282024-10-18T17:59:09.602ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52452 req_id = 5d597aec-db9a-43a7-a18e-ffefeec84584 response_code = 200 uri = /vpc-routes
29292024-10-18T17:59:09.656ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29302024-10-18T17:59:09.657ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52452 req_id = bd2b2eb2-a483-46b4-bb4a-254f855f4eb9 response_code = 204 uri = /eip-gateways
29312024-10-18T17:59:09.700ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43694
29322024-10-18T17:59:09.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43694 req_id = fb4ab594-bb28-410c-a1a8-4c37eb7daa43 response_code = 200 uri = /v2p
29332024-10-18T17:59:09.704ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52452 req_id = d3c0883f-8f26-447c-8c45-c3e119853109 response_code = 204 uri = /vpc-routes
29342024-10-18T17:59:09.715ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65460
29352024-10-18T17:59:09.715ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
29362024-10-18T17:59:09.749ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
29372024-10-18T17:59:09.802ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 87082 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:65460 req_id = 7e4943ca-b2ff-43a6-a55f-a83e6d9791ae response_code = 204 uri = /switch-ports
29382024-10-18T17:59:09.803ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65460 req_id = 014cdf78-e14f-4a76-99fc-c12152617889 response_code = 200 uri = /network-bootstore-config
29392024-10-18T17:59:10.447ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6094 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = fa90eed9-3c65-4737-ada7-988a2b5e3971 response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
29402024-10-18T17:59:22.887ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
29412024-10-18T17:59:29.056ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39590
29422024-10-18T17:59:29.056ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39590 req_id = c5a8797f-ac01-45b1-8be4-d13de1a65f54 response_code = 200 uri = /baseboard
29432024-10-18T17:59:38.750ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38956
29442024-10-18T17:59:38.751ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38956 req_id = 6103d1fc-b334-4225-80ee-e7beb1df0834 response_code = 200 uri = /vpc-routes
29452024-10-18T17:59:38.815ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29462024-10-18T17:59:38.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38956 req_id = 2c10b88c-b114-416a-b1ff-b1e7d034c1ee response_code = 204 uri = /eip-gateways
29472024-10-18T17:59:38.824ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38956 req_id = 09c419a2-2485-4af3-9bde-01b0c9c7564f response_code = 204 uri = /vpc-routes
29482024-10-18T17:59:39.071ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52563
29492024-10-18T17:59:39.071ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52563 req_id = c8f74437-57fb-4392-ae7f-332d883c4e2f response_code = 200 uri = /v2p
29502024-10-18T17:59:39.165ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48634
29512024-10-18T17:59:39.165ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
29522024-10-18T17:59:39.205ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
29532024-10-18T17:59:39.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94487 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48634 req_id = f6b68720-0895-4f73-8acf-53970696145e response_code = 204 uri = /switch-ports
29542024-10-18T17:59:39.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48634 req_id = 3af4bb0d-6456-4b99-9d4c-fe59ae0c7fe0 response_code = 200 uri = /network-bootstore-config
29552024-10-18T17:59:39.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39391
29562024-10-18T17:59:39.322ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39391 req_id = a50bc605-4090-422c-9009-aae8aa65a8ca response_code = 200 uri = /vpc-routes
29572024-10-18T17:59:39.366ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29582024-10-18T17:59:39.366ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39391 req_id = f86a9675-644f-48a9-8d07-d1150f6234bd response_code = 204 uri = /eip-gateways
29592024-10-18T17:59:39.376ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 351 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39391 req_id = 569ebecd-99e0-483e-bba4-f4ce736117fa response_code = 204 uri = /vpc-routes
29602024-10-18T17:59:39.470ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39221
29612024-10-18T17:59:39.470ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 196 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39221 req_id = c39ede56-7ad1-4946-ad30-7775d1dfac4d response_code = 200 uri = /v2p
29622024-10-18T17:59:39.545ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61165
29632024-10-18T17:59:39.545ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
29642024-10-18T17:59:39.584ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
29652024-10-18T17:59:39.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102788 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61165 req_id = 1bfa0906-d110-459c-9b9b-9b73bcb10042 response_code = 204 uri = /switch-ports
29662024-10-18T17:59:39.649ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 263 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61165 req_id = d6947eb0-54dc-4333-a610-24ecfefd676b response_code = 200 uri = /network-bootstore-config
29672024-10-18T17:59:39.750ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57502
29682024-10-18T17:59:39.750ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57502 req_id = e7b917a6-4bac-4c92-954c-b11aa2a85b28 response_code = 200 uri = /vpc-routes
29692024-10-18T17:59:39.798ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29702024-10-18T17:59:39.798ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57502 req_id = 59f9c37c-7584-4d54-bcdb-9849a214d1d9 response_code = 204 uri = /eip-gateways
29712024-10-18T17:59:39.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 447 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57502 req_id = ab523933-3429-4752-9b59-14065221e080 response_code = 204 uri = /vpc-routes
29722024-10-18T17:59:39.880ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48143
29732024-10-18T17:59:39.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48143 req_id = 1ad2d2b0-a8a4-4c06-b9fc-ec734de1ca54 response_code = 200 uri = /v2p
29742024-10-18T17:59:39.927ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37444
29752024-10-18T17:59:39.927ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
29762024-10-18T17:59:39.965ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
29772024-10-18T17:59:40.021ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94098 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37444 req_id = 711a4d09-473e-4376-8fab-5c9c429fa80e response_code = 204 uri = /switch-ports
29782024-10-18T17:59:40.022ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37444 req_id = d9ef7fef-d890-4ba1-8418-ccc31180bf1c response_code = 200 uri = /network-bootstore-config
29792024-10-18T17:59:40.531ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7248 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = bb594c00-aecd-42f4-9735-bc79a602fe37 response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
29802024-10-18T17:59:59.091ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47406
29812024-10-18T17:59:59.091ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47406 req_id = ab3cf4ce-33bb-4feb-9b27-7f09fbd43420 response_code = 200 uri = /baseboard
29822024-10-18T18:00:08.742ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60237
29832024-10-18T18:00:08.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60237 req_id = f68bb495-4135-4bee-9481-d84c1085f3e6 response_code = 200 uri = /vpc-routes
29842024-10-18T18:00:08.761ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29852024-10-18T18:00:08.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60237 req_id = 6f7e9d3a-ea35-4b7a-a463-7ef3d749fc68 response_code = 204 uri = /eip-gateways
29862024-10-18T18:00:08.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 271 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60237 req_id = 8f3f6a1b-d97b-4f5f-906c-a0e1d91babc8 response_code = 204 uri = /vpc-routes
29872024-10-18T18:00:08.801ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63819
29882024-10-18T18:00:08.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63819 req_id = d9ebc9de-e656-4694-b565-e312af04c47e response_code = 200 uri = /v2p
29892024-10-18T18:00:08.813ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49735
29902024-10-18T18:00:08.813ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
29912024-10-18T18:00:08.849ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
29922024-10-18T18:00:08.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90949 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49735 req_id = ddd55aed-760b-46e3-9f18-9576bc7de9e9 response_code = 204 uri = /switch-ports
29932024-10-18T18:00:08.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49735 req_id = 455017c7-74fa-4790-9c5b-f75814da721d response_code = 200 uri = /network-bootstore-config
29942024-10-18T18:00:09.223ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48558
29952024-10-18T18:00:09.223ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48558 req_id = 18b55269-b006-483b-92a3-6a45e8ba94cf response_code = 200 uri = /vpc-routes
29962024-10-18T18:00:09.275ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
29972024-10-18T18:00:09.275ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48558 req_id = 1bc339a1-bef7-4371-b36d-22ac8c2125e9 response_code = 204 uri = /eip-gateways
29982024-10-18T18:00:09.285ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 190 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48558 req_id = d9c49b68-04a5-48f9-a707-97f7d95939fa response_code = 204 uri = /vpc-routes
29992024-10-18T18:00:09.298ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62401
30002024-10-18T18:00:09.298ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
30012024-10-18T18:00:09.334ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
30022024-10-18T18:00:09.390ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91589 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62401 req_id = 162d2938-f8bc-4973-9a24-dec0a3f03663 response_code = 204 uri = /switch-ports
30032024-10-18T18:00:09.390ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59322
30042024-10-18T18:00:09.390ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59322 req_id = 2c611f00-a9bb-4197-bcc3-3b42fbfe4118 response_code = 200 uri = /v2p
30052024-10-18T18:00:09.390ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62401 req_id = 857fa61f-9dc1-4539-a7e8-c107896eee8f response_code = 200 uri = /network-bootstore-config
30062024-10-18T18:00:09.607ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41096
30072024-10-18T18:00:09.607ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41096 req_id = a11ef0ee-3a07-4903-9fde-3acf0c89a55e response_code = 200 uri = /vpc-routes
30082024-10-18T18:00:09.651ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
30092024-10-18T18:00:09.651ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41096 req_id = 174b5c3e-6395-4daa-8ba8-6db47f53f8f2 response_code = 204 uri = /eip-gateways
30102024-10-18T18:00:09.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 301 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41096 req_id = 4af260fb-813e-42f8-84e8-b2674a4d4019 response_code = 204 uri = /vpc-routes
30112024-10-18T18:00:09.673ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53709
30122024-10-18T18:00:09.673ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
30132024-10-18T18:00:09.713ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
30142024-10-18T18:00:09.765ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92491 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53709 req_id = 24a0a71c-2f98-4f15-8804-c654ad6e1d33 response_code = 204 uri = /switch-ports
30152024-10-18T18:00:09.765ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61271
30162024-10-18T18:00:09.766ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53709 req_id = 04697ed3-2740-42fd-911d-eacd482df65b response_code = 200 uri = /network-bootstore-config
30172024-10-18T18:00:09.766ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 258 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61271 req_id = d4de901d-9d0a-42ad-b12e-a7411c8e4618 response_code = 200 uri = /v2p
30182024-10-18T18:00:10.533ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8800 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = e8925a35-56ce-47fb-9500-08a54afefa6b response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
30192024-10-18T18:00:29.134ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55720
30202024-10-18T18:00:29.134ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55720 req_id = 33388903-e6dd-4f25-add1-93fcda3d73a4 response_code = 200 uri = /baseboard
30212024-10-18T18:00:30.852ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51396
30222024-10-18T18:00:30.852ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51396 req_id = 7ba6f804-53d5-4f8b-91b7-b7aa59a2dae7 response_code = 200 uri = /v2p
30232024-10-18T18:00:30.853ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:838 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(13955958) }
30242024-10-18T18:00:30.854ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2202 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51396 req_id = 1c8dd48a-3461-41bf-ad03-8001caee58d2 response_code = 204 uri = /v2p
30252024-10-18T18:00:31.028ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52743
30262024-10-18T18:00:31.032ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:555 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(13955958))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-f090afa56c", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 056774e0-1053-486e-ae5c-a5b6f08b5185, read_only_parent: Some(Volume { block_size: 512, id: 16e8b41b-58e9-4f5a-bd14-98a2c8b2c23c, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 474627a2-8d14-4ee0-8fde-fb4d378c6904, key: Some("HnZP228ope7abuWOUoIuif3eQr+bXrhCJvkmXNTKmSo="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::17]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 056774e0-1053-486e-ae5c-a5b6f08b5185, key: Some("2mA5nYWeIWvwngwTQ6JQg1Eb11OsMmnkg1IZfaGG/qE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::18]:19000"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-f090afa56c" }] }), cloud_init_bytes: Some(..) } instance_id = e7c1f70f-8643-4822-a729-69791d855790 metadata = InstanceMetadata { silo_id: d3065451-5d5f-40c9-9f95-bbb6046e4a73, project_id: 72f9ed17-3a48-492f-b279-5d4f818d9360 } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 2bee446a-d317-4960-b95c-6357855030de vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-18T18:00:31.028187883Z }
30272024-10-18T18:00:31.032ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:588 instance_id = e7c1f70f-8643-4822-a729-69791d855790 migration_id = None propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30282024-10-18T18:00:31.032ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1139 instance_id = e7c1f70f-8643-4822-a729-69791d855790 migration_id = None propolis_id = 2bee446a-d317-4960-b95c-6357855030de state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(13955958))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9377453b-88b2-4260-8fc5-8dfd3be4372b, kind: Instance { id: e7c1f70f-8643-4822-a729-69791d855790 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(13955958), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-f090afa56c", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 056774e0-1053-486e-ae5c-a5b6f08b5185, read_only_parent: Some(Volume { block_size: 512, id: 16e8b41b-58e9-4f5a-bd14-98a2c8b2c23c, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 474627a2-8d14-4ee0-8fde-fb4d378c6904, key: Some("HnZP228ope7abuWOUoIuif3eQr+bXrhCJvkmXNTKmSo="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::17]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 056774e0-1053-486e-ae5c-a5b6f08b5185, key: Some("2mA5nYWeIWvwngwTQ6JQg1Eb11OsMmnkg1IZfaGG/qE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::18]:19000"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-f090afa56c" }] }), cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-18T18:00:31.028187883Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
30292024-10-18T18:00:31.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6347 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52743 req_id = 8cb0a2e3-1500-485e-824e-4d14ec4bfdd2 response_code = 200 uri = /vmms/2bee446a-d317-4960-b95c-6357855030de
30302024-10-18T18:00:31.146ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40657
30312024-10-18T18:00:31.147ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 181 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40657 req_id = 131fc7be-e5b8-4964-b770-eebf3e3b6db0 response_code = 200 uri = /vmms/2bee446a-d317-4960-b95c-6357855030de/state
30322024-10-18T18:00:31.148ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 13955958 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
30332024-10-18T18:00:31.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
30342024-10-18T18:00:31.190ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de file = illumos-utils/src/zone.rs:316 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30352024-10-18T18:00:31.214ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de file = illumos-utils/src/zone.rs:354 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30362024-10-18T18:00:31.269ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60146
30372024-10-18T18:00:31.269ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60146 req_id = b8d2331c-d7e7-401c-949e-cea5d6cc33bd response_code = 200 uri = /vpc-routes
30382024-10-18T18:00:31.290ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {9377453b-88b2-4260-8fc5-8dfd3be4372b: {10.151.1.176: {3d12df86-6d6b-46e5-b65a-0371b86694f6}, 10.151.1.175: {3d12df86-6d6b-46e5-b65a-0371b86694f6}}} }
30392024-10-18T18:00:32.475ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30402024-10-18T18:00:32.476ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de zone file = illumos-utils/src/running_zone.rs:488 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30412024-10-18T18:00:32.652ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 40.124176ms file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30422024-10-18T18:00:32.701ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 69.809172ms file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30432024-10-18T18:00:32.785ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 187.780076ms file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30442024-10-18T18:00:32.991ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 534.121321ms file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30452024-10-18T18:00:33.541ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 607.815054ms file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30462024-10-18T18:00:34.161ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 956.211316ms file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30472024-10-18T18:00:35.128ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 1.425613843s file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30482024-10-18T18:00:36.565ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 529.144862ms file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30492024-10-18T18:00:37.104ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de") failed: Property not found. retry in 1.393209594s file = illumos-utils/src/svc.rs:36 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de zone = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
30502024-10-18T18:00:38.533ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de file = sled-agent/src/instance.rs:1633 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30512024-10-18T18:00:38.545ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1642 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30522024-10-18T18:00:38.589ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1666 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30532024-10-18T18:00:38.731ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:835 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30542024-10-18T18:00:38.732ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1205 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
30552024-10-18T18:00:38.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7442607 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60146 req_id = c783ef85-a35f-45e9-8dcb-26e6d95f3a32 response_code = 204 uri = /eip-gateways
30562024-10-18T18:00:38.733ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-10-18T18:00:38.733167593Z } file = sled-agent/src/instance.rs:685 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30572024-10-18T18:00:38.733ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = e7c1f70f-8643-4822-a729-69791d855790 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-18T18:00:38.733167593Z } propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30582024-10-18T18:00:38.733ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-18T18:00:38.733167593Z }, migration_in: None, migration_out: None }
30592024-10-18T18:00:38.842ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57785
30602024-10-18T18:00:38.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 46186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57785 req_id = 2d702cde-834a-4b5a-813a-b5517dce2c94 response_code = 200 uri = /vmms/2bee446a-d317-4960-b95c-6357855030de/state
30612024-10-18T18:00:38.893ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-10-18T18:00:38.893669079Z } file = sled-agent/src/instance.rs:685 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30622024-10-18T18:00:38.893ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = e7c1f70f-8643-4822-a729-69791d855790 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-18T18:00:38.893669079Z } propolis_id = 2bee446a-d317-4960-b95c-6357855030de
30632024-10-18T18:00:38.893ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-18T18:00:38.893669079Z }, migration_in: None, migration_out: None }
30642024-10-18T18:00:39.130ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39576
30652024-10-18T18:00:39.130ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
30662024-10-18T18:00:39.176ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
30672024-10-18T18:00:39.239ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109570 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39576 req_id = c183e0d7-02fc-4379-9ced-fab1c00cec2c response_code = 204 uri = /switch-ports
30682024-10-18T18:00:39.240ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62257
30692024-10-18T18:00:39.240ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39576 req_id = 1fff0ca4-f859-42e7-9521-6051cbbee147 response_code = 200 uri = /network-bootstore-config
30702024-10-18T18:00:39.240ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 289 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62257 req_id = dad74132-38cf-4f2d-bc11-0560db65fd80 response_code = 200 uri = /v2p
30712024-10-18T18:00:39.241ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 876 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60146 req_id = b7a01c3b-5135-4266-94ae-8e2e3b1fcdd2 response_code = 204 uri = /vpc-routes
30722024-10-18T18:00:39.340ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58986
30732024-10-18T18:00:39.340ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58986 req_id = 089cd258-4710-4389-b724-37bbd523988e response_code = 200 uri = /vpc-routes
30742024-10-18T18:00:39.385ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {9377453b-88b2-4260-8fc5-8dfd3be4372b: {10.151.1.176: {3d12df86-6d6b-46e5-b65a-0371b86694f6}, 10.151.1.175: {3d12df86-6d6b-46e5-b65a-0371b86694f6}}} }
30752024-10-18T18:00:39.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1051 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58986 req_id = 779a148b-e2a9-4464-97c2-ab17c87ca129 response_code = 204 uri = /eip-gateways
30762024-10-18T18:00:39.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 256 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58986 req_id = 042edeb7-0152-4744-ba6a-8e2c513f8d13 response_code = 204 uri = /vpc-routes
30772024-10-18T18:00:39.406ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39491
30782024-10-18T18:00:39.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 34 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39491 req_id = 9cebd55b-1c85-463f-97d0-2624f6cb61be response_code = 200 uri = /vpc-routes
30792024-10-18T18:00:39.484ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34469
30802024-10-18T18:00:39.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34469 req_id = 92aa0bd1-26aa-445a-9d68-013318c87b8a response_code = 200 uri = /vmms/2bee446a-d317-4960-b95c-6357855030de/state
30812024-10-18T18:00:39.565ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38243
30822024-10-18T18:00:39.565ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38243 req_id = 7955b5a6-0376-4e40-9b30-6e0bc6c468f7 response_code = 200 uri = /v2p
30832024-10-18T18:00:39.614ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {9377453b-88b2-4260-8fc5-8dfd3be4372b: {10.151.1.175: {3d12df86-6d6b-46e5-b65a-0371b86694f6}, 10.151.1.176: {3d12df86-6d6b-46e5-b65a-0371b86694f6}}} }
30842024-10-18T18:00:39.614ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39491 req_id = d07e5d0b-6a41-42e5-9d16-311be2786ba3 response_code = 204 uri = /eip-gateways
30852024-10-18T18:00:39.631ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 251 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39491 req_id = d67686d9-eae3-48e7-88f0-75526c4af0d6 response_code = 204 uri = /vpc-routes
30862024-10-18T18:00:39.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35312
30872024-10-18T18:00:39.690ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
30882024-10-18T18:00:39.733ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
30892024-10-18T18:00:39.796ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 105783 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35312 req_id = d952c38c-8ddb-4693-aa43-afca7d409d88 response_code = 204 uri = /switch-ports
30902024-10-18T18:00:39.796ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45486
30912024-10-18T18:00:39.796ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54374
30922024-10-18T18:00:39.796ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45486 req_id = ab315f49-3631-45bc-a010-3673a4402221 response_code = 200 uri = /vpc-routes
30932024-10-18T18:00:39.796ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 288 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35312 req_id = a9dd1c6b-f820-40ea-820b-870c0d45b969 response_code = 200 uri = /network-bootstore-config
30942024-10-18T18:00:39.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 239 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54374 req_id = 6173d1cc-8055-43dc-b8d8-215973c94d6f response_code = 200 uri = /vmms/2bee446a-d317-4960-b95c-6357855030de/state
30952024-10-18T18:00:39.813ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {9377453b-88b2-4260-8fc5-8dfd3be4372b: {10.151.1.175: {3d12df86-6d6b-46e5-b65a-0371b86694f6}, 10.151.1.176: {3d12df86-6d6b-46e5-b65a-0371b86694f6}}} }
30962024-10-18T18:00:39.813ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45486 req_id = 383503f6-a03a-4411-bd03-dc419fb1f12b response_code = 204 uri = /eip-gateways
30972024-10-18T18:00:39.936ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35248
30982024-10-18T18:00:39.936ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35248 req_id = 08daaa7e-b9ff-4c2e-b170-5fa054df3ebc response_code = 200 uri = /v2p
30992024-10-18T18:00:39.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 802 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45486 req_id = 712ae181-0679-4a3c-b889-36a6dc9dd680 response_code = 204 uri = /vpc-routes
31002024-10-18T18:00:40.024ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52088
31012024-10-18T18:00:40.024ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
31022024-10-18T18:00:40.061ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
31032024-10-18T18:00:40.114ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89981 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52088 req_id = 08afc67d-c357-41bb-aed1-9745925109d1 response_code = 204 uri = /switch-ports
31042024-10-18T18:00:40.115ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 227 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52088 req_id = 7c92b2aa-cce6-4fe7-98df-78d1912c53b3 response_code = 200 uri = /network-bootstore-config
31052024-10-18T18:00:40.532ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7607 local_addr = [fd00:1122:3344:101::1]:65402 method = GET remote_addr = [fd00:1122:3344:101::d]:37310 req_id = e6e59a1b-3fc3-4d56-80c4-21246e069042 response_code = 200 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 uri = /08066c3d-b7ed-49d2-8578-f8769fb94736
31062024-10-18T18:00:48.671ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46637
31072024-10-18T18:00:48.671ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 680 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46637 req_id = 8fb6b264-f36f-4739-81db-1e12cb607379 response_code = 200 uri = /vmms/2bee446a-d317-4960-b95c-6357855030de/state
31082024-10-18T18:00:48.672ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-10-18T18:00:48.672109261Z } file = sled-agent/src/instance.rs:685 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31092024-10-18T18:00:48.672ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = e7c1f70f-8643-4822-a729-69791d855790 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-18T18:00:48.672109261Z } propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31102024-10-18T18:00:48.672ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-18T18:00:48.672109261Z }, migration_in: None, migration_out: None }
31112024-10-18T18:00:48.709ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-10-18T18:00:48.709757709Z } file = sled-agent/src/instance.rs:685 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31122024-10-18T18:00:48.709ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = e7c1f70f-8643-4822-a729-69791d855790 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-10-18T18:00:48.709757709Z } propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31132024-10-18T18:00:48.709ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:721 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31142024-10-18T18:00:48.709ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
31152024-10-18T18:00:48.709ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 4431706173448737589 n_samples = 6 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
31162024-10-18T18:00:48.709ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 14398483284864988566 n_samples = 6 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
31172024-10-18T18:00:48.712ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
31182024-10-18T18:00:48.713ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/62298c00-fb90-4e92-a4b4-95bca961978c/debug/bundle/zone", "/pool/int/e40afb95-7af8-4f62-aab4-e63b3bd02855/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/8853a1c1-d102-4f80-9012-356473e594ff/crypt/debug", "/pool/ext/8523f2c5-b045-4f20-8f22-89f361c6e33c/crypt/debug", "/pool/ext/24197587-83af-46c0-a36e-5e4a26350d21/crypt/debug", "/pool/ext/956d8649-5e02-4c83-a766-709d6c1c19f4/crypt/debug", "/pool/ext/4e3d3772-4d1a-4cd4-9451-d3f5402e0376/crypt/debug", "/pool/ext/aa3e117f-01a2-4173-8076-71ed9651de49/crypt/debug", "/pool/ext/e09cb849-5302-41dd-87a5-36e8723c48fb/crypt/debug", "/pool/ext/c6223f26-38f9-4483-9d17-efd8e60dd490/crypt/debug", "/pool/ext/883d1cbf-e1f8-4c86-b18f-d626feb923fd/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de
31192024-10-18T18:00:49.366ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de", bundle_id: 031967eb-ca93-40f9-8a1a-b22d7b894461 }, time_created: 2024-10-18T18:00:48.714112748Z, version: 0, cause: TerminatedInstance }
31202024-10-18T18:00:49.366ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_2bee446a-d317-4960-b95c-6357855030de file = sled-agent/src/instance.rs:935 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31212024-10-18T18:00:51.107ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31222024-10-18T18:00:51.123ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(13955958), kind: System }
31232024-10-18T18:00:51.123ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(13955958), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
31242024-10-18T18:00:51.124ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-18T18:00:51.124082774Z }, migration_in: None, migration_out: None }
31252024-10-18T18:00:51.146ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-18T18:00:51.124082774Z }, migration_in: None, migration_out: None }
31262024-10-18T18:00:51.150ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:872 instance_id = e7c1f70f-8643-4822-a729-69791d855790 propolis_id = 2bee446a-d317-4960-b95c-6357855030de
31272024-10-18T18:00:51.599ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52505
31282024-10-18T18:00:51.599ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52505 req_id = 651949d7-757d-4439-b6a9-ef03d7fa0781 response_code = 200 uri = /vpc-routes
31292024-10-18T18:00:51.613ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {9377453b-88b2-4260-8fc5-8dfd3be4372b: {10.151.1.176: {3d12df86-6d6b-46e5-b65a-0371b86694f6}, 10.151.1.175: {3d12df86-6d6b-46e5-b65a-0371b86694f6}}} }
31302024-10-18T18:00:51.613ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52505 req_id = 14319037-f41a-413c-b365-8e24cb528708 response_code = 204 uri = /eip-gateways
31312024-10-18T18:00:51.622ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 262 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52505 req_id = c12ef76d-7637-47fd-8f15-c8bc1e6fd268 response_code = 204 uri = /vpc-routes
31322024-10-18T18:00:51.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47812
31332024-10-18T18:00:51.730ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47812 req_id = b6369476-aeec-4374-becd-df74c7d824a6 response_code = 200 uri = /v2p
31342024-10-18T18:00:52.206ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54993
31352024-10-18T18:00:52.207ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54993 req_id = 508b3def-2a91-410f-8439-3041ba19f7b0 response_code = 200 uri = /v2p
31362024-10-18T18:00:52.295ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63711
31372024-10-18T18:00:52.295ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63711 req_id = 5c6d5bfd-81d4-4629-bbd7-90b613b46c85 response_code = 200 uri = /vpc-routes
31382024-10-18T18:00:52.319ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
31392024-10-18T18:00:52.319ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1205 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736
31402024-10-18T18:00:52.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63711 req_id = b2bbc819-b7b9-4c15-aab8-44e095f048b9 response_code = 204 uri = /eip-gateways
31412024-10-18T18:00:52.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 338 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63711 req_id = c1f3dde6-c965-4107-9a5c-6dbe7c8167ee response_code = 204 uri = /vpc-routes
31422024-10-18T18:00:53.031ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34568
31432024-10-18T18:00:53.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34568 req_id = 00af5926-491d-4d49-b990-82ea3650129e response_code = 200 uri = /vpc-routes
31442024-10-18T18:00:53.046ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = 08066c3d-b7ed-49d2-8578-f8769fb94736 values = ExternalIpGatewayMap { mappings: {} }
31452024-10-18T18:00:53.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34568 req_id = 51de356b-aee9-4f7e-8930-f4d02544692c response_code = 204 uri = /eip-gateways
31462024-10-18T18:00:53.054ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 170 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34568 req_id = 90af21f7-3db9-4a7f-a28e-040206fbf608 response_code = 204 uri = /vpc-routes