Max level shown:
1 [ Feb 13 05:05:32 Enabled. ]
2 [ Feb 13 05:05:32 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Feb 13 05:05:32 Method "start" exited with status 0. ]
4 [ Feb 13 05:05:32 Rereading configuration. ]
5 [ Feb 13 05:05:32 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72025-02-13T05:05:32.126ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82025-02-13T05:05:32.127ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92025-02-13T05:05:32.179ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102025-02-13T05:05:32.180ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112025-02-13T05:05:32.296ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122025-02-13T05:05:32.306ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132025-02-13T05:05:32.315ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142025-02-13T05:05:32.920ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152025-02-13T05:05:33.058ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162025-02-13T05:05:33.058ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172025-02-13T05:05:33.058ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182025-02-13T05:05:33.058ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
192025-02-13T05:05:33.058ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
202025-02-13T05:05:33.058ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212025-02-13T05:05:33.061ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222025-02-13T05:05:33.061ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232025-02-13T05:05:33.061ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242025-02-13T05:05:33.061ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252025-02-13T05:05:33.061ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262025-02-13T05:05:33.062ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272025-02-13T05:05:33.062ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
282025-02-13T05:05:33.062ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292025-02-13T05:05:33.062ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302025-02-13T05:05:33.073ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312025-02-13T05:05:33.073ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool)
322025-02-13T05:05:33.172ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal }
332025-02-13T05:05:33.542ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:607
342025-02-13T05:05:33.542ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352025-02-13T05:05:33.542ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362025-02-13T05:05:33.542ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372025-02-13T05:05:33.542ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
382025-02-13T05:05:33.542ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392025-02-13T05:05:33.542ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402025-02-13T05:05:33.551ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412025-02-13T05:05:33.551ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool)
422025-02-13T05:05:33.640ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal }
432025-02-13T05:05:34.001ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442025-02-13T05:05:34.001ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
472025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
482025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
492025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
512025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
522025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
602025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
662025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
722025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
782025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
822025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
832025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
842025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
872025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912025-02-13T05:05:34.002ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
932025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
942025-02-13T05:05:34.002ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
952025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972025-02-13T05:05:34.002ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982025-02-13T05:05:34.002ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
992025-02-13T05:05:34.002ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
1002025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012025-02-13T05:05:34.002ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022025-02-13T05:05:34.002ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1032025-02-13T05:05:34.002ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042025-02-13T05:05:34.002ZINFOSledAgent (bootstore): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052025-02-13T05:05:34.002ZINFOSledAgent (bootstore): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062025-02-13T05:05:34.002ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072025-02-13T05:05:34.002ZINFOSledAgent (bootstore): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082025-02-13T05:05:34.002ZINFOSledAgent (bootstore): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092025-02-13T05:05:34.002ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102025-02-13T05:05:34.002ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112025-02-13T05:05:34.002ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122025-02-13T05:05:34.013ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:794
1132025-02-13T05:05:34.013ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1243, tv_nsec: 423514571 } time_to_next_cleanup = 599.999908137s
1142025-02-13T05:05:34.013ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1152025-02-13T05:05:34.013ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:4019
1162025-02-13T05:05:34.013ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/sled-agent-request.json file = common/src/ledger.rs:188
1172025-02-13T05:05:34.013ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/sled-agent-request.json file = common/src/ledger.rs:188
1182025-02-13T05:05:34.016ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:328 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192025-02-13T05:05:34.018ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202025-02-13T05:05:34.019ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: Some( None, ), bgp_peers: [], autoneg: false, lldp: None, tx_eq: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1195
1212025-02-13T05:05:34.019ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1222025-02-13T05:05:34.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1232025-02-13T05:05:34.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1242025-02-13T05:05:34.019ZINFOSledAgent (RSS): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/rss-started.marker file = common/src/ledger.rs:188
1252025-02-13T05:05:34.019ZINFOSledAgent (RSS): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/rss-started.marker file = common/src/ledger.rs:188
1262025-02-13T05:05:34.019ZINFOSledAgent (RSS): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1272025-02-13T05:05:34.019ZINFOSledAgent (RSS): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1282025-02-13T05:05:34.019ZINFOSledAgent (RSS): RSS not previously run. Creating plans. file = sled-agent/src/rack_setup/service.rs:1244
1292025-02-13T05:05:34.019ZINFOSledAgent (RSS): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.rss-started.marker.tmp file = common/src/ledger.rs:199
1302025-02-13T05:05:34.019ZINFOSledAgent (RSS): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.rss-started.marker.tmp file = common/src/ledger.rs:199
1312025-02-13T05:05:34.019ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1276
1322025-02-13T05:05:34.019ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:66
1332025-02-13T05:05:34.019ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:39
1342025-02-13T05:05:34.019ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1307
1352025-02-13T05:05:34.019ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1362025-02-13T05:05:34.019ZINFOSledAgent (bootstore): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1372025-02-13T05:05:34.020ZINFOSledAgent (bootstore): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1382025-02-13T05:05:34.020ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1392025-02-13T05:05:34.020ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 5835b48f-f45d-4fa8-91ed-38d933d73732 (sled), rack_id: c3272b8c-876d-43d9-a1f4-8963a7553b1d, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1402025-02-13T05:05:34.020ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4316
1412025-02-13T05:05:34.020ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4722
1422025-02-13T05:05:34.025ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1432025-02-13T05:05:34.031ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1442025-02-13T05:05:34.031ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1452025-02-13T05:05:34.035ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462025-02-13T05:05:34.035ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472025-02-13T05:05:34.036ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:a02a::1]:46840
1482025-02-13T05:05:34.036ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 5835b48f-f45d-4fa8-91ed-38d933d73732 (sled), rack_id: c3272b8c-876d-43d9-a1f4-8963a7553b1d, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1492025-02-13T05:05:34.036ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1502025-02-13T05:05:34.036ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1512025-02-13T05:05:34.036ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1522025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:607
1532025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:813
1542025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1552025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1562025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:802
1572025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:830
1582025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:596
1592025-02-13T05:05:34.036ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1602025-02-13T05:05:34.036ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1612025-02-13T05:05:34.036ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:701 result = DisksManagementResult { status: [] }
1622025-02-13T05:05:34.051ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1632025-02-13T05:05:34.071ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:325
1642025-02-13T05:05:34.088ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:363
1652025-02-13T05:05:34.098ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:419 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1662025-02-13T05:05:34.098ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1672025-02-13T05:05:34.098ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1682025-02-13T05:05:34.098ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:431 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1692025-02-13T05:05:34.101ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 124.48733ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9905211a-ed90-4202-98e1-b2b0667c1c2e", "content-length": "124", "date": "Thu, 13 Feb 2025 05:05:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9905211a-ed90-4202-98e1-b2b0667c1c2e" } file = clients/ddm-admin-client/src/lib.rs:177
1702025-02-13T05:05:34.113ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_db71a81c-96cf-41f7-8c27-fbb635fcb5d5/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1712025-02-13T05:05:34.136ZINFOSledAgent: successfully created encrypted zvol: name="oxi_db71a81c-96cf-41f7-8c27-fbb635fcb5d5/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1722025-02-13T05:05:34.136ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_db71a81c-96cf-41f7-8c27-fbb635fcb5d5/swap" file = sled-agent/src/swap_device.rs:96
1732025-02-13T05:05:34.141ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:446 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1742025-02-13T05:05:34.141ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:129
1752025-02-13T05:05:34.141ZINFOSledAgent (BackingFs): Ensuring dataset oxi_db71a81c-96cf-41f7-8c27-fbb635fcb5d5/backing/fmd file = sled-agent/src/backing_fs.rs:139
1762025-02-13T05:05:34.209ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:174
1772025-02-13T05:05:34.227ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1782025-02-13T05:05:34.227ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 139.363152ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b8f88fba-6b74-4cb5-9d2e-54463bf71a61", "content-length": "124", "date": "Thu, 13 Feb 2025 05:05:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b8f88fba-6b74-4cb5-9d2e-54463bf71a61" } file = clients/ddm-admin-client/src/lib.rs:177
1792025-02-13T05:05:34.368ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1802025-02-13T05:05:34.368ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 97.70354ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "13c2a0f0-1263-4f1c-b03f-7c8a2609952f", "content-length": "124", "date": "Thu, 13 Feb 2025 05:05:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "13c2a0f0-1263-4f1c-b03f-7c8a2609952f" } file = clients/ddm-admin-client/src/lib.rs:177
1812025-02-13T05:05:34.427ZINFOSledAgent (BackingFs): Mounting oxi_db71a81c-96cf-41f7-8c27-fbb635fcb5d5/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:182
1822025-02-13T05:05:34.440ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:190
1832025-02-13T05:05:34.440ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:190
1842025-02-13T05:05:34.440ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:190
1852025-02-13T05:05:34.440ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:196
1862025-02-13T05:05:34.467ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1872025-02-13T05:05:34.468ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 221.706264ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c800728e-df93-400e-afce-c0fd65338bdd", "content-length": "124", "date": "Thu, 13 Feb 2025 05:05:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c800728e-df93-400e-afce-c0fd65338bdd" } file = clients/ddm-admin-client/src/lib.rs:177
1882025-02-13T05:05:34.691ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892025-02-13T05:05:34.692ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 240.415415ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b05a1084-d384-49bf-bb58-2479bf6db0d8", "content-length": "124", "date": "Thu, 13 Feb 2025 05:05:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b05a1084-d384-49bf-bb58-2479bf6db0d8" } file = clients/ddm-admin-client/src/lib.rs:177
1902025-02-13T05:05:34.933ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912025-02-13T05:05:34.934ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 319.352099ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ef11ab3c-6477-4927-8fb7-2750c205a46c", "content-length": "124", "date": "Thu, 13 Feb 2025 05:05:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ef11ab3c-6477-4927-8fb7-2750c205a46c" } file = clients/ddm-admin-client/src/lib.rs:177
1922025-02-13T05:05:35.174ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1932025-02-13T05:05:35.180ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:328 local_addr = [fd00:1122:3344:101::1]:40836 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1942025-02-13T05:05:35.180ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1952025-02-13T05:05:35.180ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:40836 file = oximeter/producer/src/lib.rs:279 interval = 30s producer_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1962025-02-13T05:05:35.181ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 311.944956ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1972025-02-13T05:05:35.182ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1982025-02-13T05:05:35.182ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
1992025-02-13T05:05:35.182ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:978 underlay_address = fd00:1122:3344:101::1
2002025-02-13T05:05:35.182ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
2012025-02-13T05:05:35.254ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2022025-02-13T05:05:35.494ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 419.248028ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2032025-02-13T05:05:35.916ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.051682513s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2042025-02-13T05:05:36.970ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.625472767s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2052025-02-13T05:05:37.853ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2750
2062025-02-13T05:05:37.853ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2978
2072025-02-13T05:05:37.853ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2620
2082025-02-13T05:05:37.853ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2667
2092025-02-13T05:05:37.853ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:3059
2102025-02-13T05:05:37.853ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3104
2112025-02-13T05:05:37.855ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2886
2122025-02-13T05:05:37.857ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2661
2132025-02-13T05:05:37.857ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='68ff15a7-efff-4e3b-9b70-a8c9f3b33a89'/> <propval type="astring" name="rack_id" value='c3272b8c-876d-43d9-a1f4-8963a7553b1d'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe1d:2a7a'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="rack_id" value='c3272b8c-876d-43d9-a1f4-8963a7553b1d'/> <propval type="astring" name="sled_id" value='5835b48f-f45d-4fa8-91ed-38d933d73732'/> <propval type="astring" name="sled_model" value='i86pc'/> <propval type="astring" name="sled_serial" value='sock'/> <propval type="astring" name="sled_revision" value='0'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dpd_host" value='[::1]'/> <propval type="astring" name="dpd_port" value='12224'/> <propval type="astring" name="rack_id" value='c3272b8c-876d-43d9-a1f4-8963a7553b1d'/> <propval type="astring" name="sled_id" value='5835b48f-f45d-4fa8-91ed-38d933d73732'/> <propval type="astring" name="sled_model" value='i86pc'/> <propval type="astring" name="sled_serial" value='sock'/> <propval type="astring" name="sled_revision" value='0'/> <propval type="astring" name="listen_address" value='[::1]:12231'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='5835b48f-f45d-4fa8-91ed-38d933d73732'/> <propval type="astring" name="rack_uuid" value='c3272b8c-876d-43d9-a1f4-8963a7553b1d'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='5835b48f-f45d-4fa8-91ed-38d933d73732'/> <propval type="astring" name="rack_uuid" value='c3272b8c-876d-43d9-a1f4-8963a7553b1d'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2142025-02-13T05:05:37.857ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_switch
2152025-02-13T05:05:38.123ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_switch") failed: Property not found. retry in 53.039515ms file = illumos-utils/src/svc.rs:42 zone = oxz_switch
2162025-02-13T05:05:38.347ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_switch") failed: Property not found. retry in 142.629207ms file = illumos-utils/src/svc.rs:42 zone = oxz_switch
2172025-02-13T05:05:38.523ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_switch") failed: Property not found. retry in 206.718245ms file = illumos-utils/src/svc.rs:42 zone = oxz_switch
2182025-02-13T05:05:38.853ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_switch") failed: Property not found. retry in 508.768944ms file = illumos-utils/src/svc.rs:42 zone = oxz_switch
2192025-02-13T05:05:39.587ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_switch") failed: Property not found. retry in 996.905508ms file = illumos-utils/src/svc.rs:42 zone = oxz_switch
2202025-02-13T05:05:39.596ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.890194048s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2212025-02-13T05:05:40.769ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_switch") failed: Property not found. retry in 1.416279498s file = illumos-utils/src/svc.rs:42 zone = oxz_switch
2222025-02-13T05:05:42.343ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_switch") failed: Property not found. retry in 1.290888174s file = illumos-utils/src/svc.rs:42 zone = oxz_switch
2232025-02-13T05:05:44.025ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2242025-02-13T05:05:44.026ZINFOSledAgent (dropshot (Repo Depot)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:328 local_addr = [fd00:1122:3344:101::1]:12348 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2252025-02-13T05:05:44.027ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:328 local_addr = [fd00:1122:3344:101::1]:12345
2262025-02-13T05:05:44.027ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2272025-02-13T05:05:44.027ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2282025-02-13T05:05:44.027ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2292025-02-13T05:05:44.027ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2302025-02-13T05:05:44.027ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2312025-02-13T05:05:44.028ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: Some(None), bgp_peers: [], autoneg: false, lldp: None, tx_eq: None }], bgp: [], bfd: [] })))
2322025-02-13T05:05:44.028ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:4019
2332025-02-13T05:05:44.028ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:a02a::1]:46840
2342025-02-13T05:05:44.270ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2352025-02-13T05:05:44.270ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2362025-02-13T05:05:44.275ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4340 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2372025-02-13T05:05:44.275ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4356
2382025-02-13T05:05:44.275ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:535 zone = oxz_switch
2392025-02-13T05:05:44.553ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 11.327100045s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2402025-02-13T05:05:44.586ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51782
2412025-02-13T05:05:44.586ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2422025-02-13T05:05:44.586ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2432025-02-13T05:05:44.586ZINFOSledAgent (ServiceManager): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:188
2442025-02-13T05:05:44.586ZINFOSledAgent (ServiceManager): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:188
2452025-02-13T05:05:44.587ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 307 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51782 req_id = d211a69f-e190-428e-8c0f-0524d505b9db response_code = 200 uri = /inventory
2462025-02-13T05:05:44.617ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33339
2472025-02-13T05:05:44.617ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33339 req_id = c35773c1-4e42-4dc7-94ef-3dc7b957470e response_code = 200 uri = /sled-role
2482025-02-13T05:05:44.648ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(2), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 67a4db0f-6892-49b7-8b66-559ccce18ef4 (physical_disk), pool_id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 77dd649c-a667-4ebb-ac67-cff96405e1c9 (physical_disk), pool_id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 83139c92-b163-4047-833e-274061c2ea28 (physical_disk), pool_id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 897a5f96-a409-47a3-b1c1-a1509a18f07c (physical_disk), pool_id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: a17a4e2c-6456-4908-86cd-157cbbd1ec7e (physical_disk), pool_id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: c3f0cc78-0aa0-4a13-bdef-7616e977ff2a (physical_disk), pool_id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: cb26102f-eeb7-4f8b-805b-d03511ce3ed0 (physical_disk), pool_id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: da4aef87-50df-4739-b731-8276531b7e8a (physical_disk), pool_id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: dd287d72-7bbc-483b-b87c-b8c5570cf8db (physical_disk), pool_id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:385 sled_address = [fd00:1122:3344:101::1]:12345
2492025-02-13T05:05:44.648ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45474
2502025-02-13T05:05:44.649ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:895 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2512025-02-13T05:05:44.649ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(2), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 67a4db0f-6892-49b7-8b66-559ccce18ef4 (physical_disk), pool_id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 77dd649c-a667-4ebb-ac67-cff96405e1c9 (physical_disk), pool_id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 83139c92-b163-4047-833e-274061c2ea28 (physical_disk), pool_id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 897a5f96-a409-47a3-b1c1-a1509a18f07c (physical_disk), pool_id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: a17a4e2c-6456-4908-86cd-157cbbd1ec7e (physical_disk), pool_id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: c3f0cc78-0aa0-4a13-bdef-7616e977ff2a (physical_disk), pool_id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: cb26102f-eeb7-4f8b-805b-d03511ce3ed0 (physical_disk), pool_id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: da4aef87-50df-4739-b731-8276531b7e8a (physical_disk), pool_id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: dd287d72-7bbc-483b-b87c-b8c5570cf8db (physical_disk), pool_id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:607
2522025-02-13T05:05:44.649ZINFOSledAgent (StorageManager): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2532025-02-13T05:05:44.649ZINFOSledAgent (StorageManager): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2542025-02-13T05:05:44.649ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:1298 request = omicron_physical_disks_ensure
2552025-02-13T05:05:44.649ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2562025-02-13T05:05:44.649ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2572025-02-13T05:05:44.649ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2582025-02-13T05:05:44.649ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2592025-02-13T05:05:44.788ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2602025-02-13T05:05:44.788ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool)
2612025-02-13T05:05:45.029ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2622025-02-13T05:05:45.229ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11041192580377101784 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2632025-02-13T05:05:45.229ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14879566039012961736 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2642025-02-13T05:05:45.443ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
2652025-02-13T05:05:45.492ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }
2662025-02-13T05:05:45.500ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:239
2672025-02-13T05:05:45.500ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:241
2682025-02-13T05:05:45.500ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:245
2692025-02-13T05:05:45.501ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:247
2702025-02-13T05:05:45.501ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2712025-02-13T05:05:45.501ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2722025-02-13T05:05:45.501ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
2732025-02-13T05:05:45.536ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2742025-02-13T05:05:45.544ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/zone file = sled-storage/src/dataset.rs:302
2752025-02-13T05:05:45.683ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 file = sled-storage/src/dataset.rs:388
2762025-02-13T05:05:45.691ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2772025-02-13T05:05:45.691ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2782025-02-13T05:05:45.691ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2792025-02-13T05:05:45.691ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2802025-02-13T05:05:45.691ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2812025-02-13T05:05:45.700ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2822025-02-13T05:05:45.700ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 92d599bb-7067-4926-9301-d5f5f983f111 (zpool)
2832025-02-13T05:05:45.797ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }
2842025-02-13T05:05:45.805ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:239
2852025-02-13T05:05:45.805ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:241
2862025-02-13T05:05:45.805ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:245
2872025-02-13T05:05:45.805ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:247
2882025-02-13T05:05:45.805ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2892025-02-13T05:05:45.805ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2902025-02-13T05:05:45.805ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
2912025-02-13T05:05:45.839ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2922025-02-13T05:05:45.847ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_92d599bb-7067-4926-9301-d5f5f983f111/crypt/zone file = sled-storage/src/dataset.rs:302
2932025-02-13T05:05:45.986ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_92d599bb-7067-4926-9301-d5f5f983f111 file = sled-storage/src/dataset.rs:388
2942025-02-13T05:05:45.994ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
2952025-02-13T05:05:45.994ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
2962025-02-13T05:05:45.994ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
2972025-02-13T05:05:45.994ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2982025-02-13T05:05:45.994ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
2992025-02-13T05:05:46.004ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3002025-02-13T05:05:46.004ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool)
3012025-02-13T05:05:46.102ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }
3022025-02-13T05:05:46.110ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:239
3032025-02-13T05:05:46.110ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:241
3042025-02-13T05:05:46.110ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:245
3052025-02-13T05:05:46.110ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:247
3062025-02-13T05:05:46.110ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3072025-02-13T05:05:46.111ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3082025-02-13T05:05:46.111ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3092025-02-13T05:05:46.147ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3102025-02-13T05:05:46.151ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
3112025-02-13T05:05:46.155ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/zone file = sled-storage/src/dataset.rs:302
3122025-02-13T05:05:46.295ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9488b08f-08d1-4217-90af-27a0f69b51f7 file = sled-storage/src/dataset.rs:388
3132025-02-13T05:05:46.304ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3142025-02-13T05:05:46.304ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3152025-02-13T05:05:46.304ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3162025-02-13T05:05:46.304ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3172025-02-13T05:05:46.304ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3182025-02-13T05:05:46.314ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3192025-02-13T05:05:46.314ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool)
3202025-02-13T05:05:46.418ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }
3212025-02-13T05:05:46.426ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:239
3222025-02-13T05:05:46.426ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:241
3232025-02-13T05:05:46.426ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:245
3242025-02-13T05:05:46.426ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:247
3252025-02-13T05:05:46.426ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3262025-02-13T05:05:46.426ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3272025-02-13T05:05:46.426ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3282025-02-13T05:05:46.462ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3292025-02-13T05:05:46.470ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone file = sled-storage/src/dataset.rs:302
3302025-02-13T05:05:46.609ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 file = sled-storage/src/dataset.rs:388
3312025-02-13T05:05:46.617ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3322025-02-13T05:05:46.617ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3332025-02-13T05:05:46.617ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3342025-02-13T05:05:46.617ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3352025-02-13T05:05:46.617ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3362025-02-13T05:05:46.627ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3372025-02-13T05:05:46.627ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool)
3382025-02-13T05:05:46.717ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }
3392025-02-13T05:05:46.726ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:239
3402025-02-13T05:05:46.726ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:241
3412025-02-13T05:05:46.726ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:245
3422025-02-13T05:05:46.726ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:247
3432025-02-13T05:05:46.726ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3442025-02-13T05:05:46.726ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3452025-02-13T05:05:46.726ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3462025-02-13T05:05:46.760ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3472025-02-13T05:05:46.768ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/zone file = sled-storage/src/dataset.rs:302
3482025-02-13T05:05:46.904ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_15d1957c-72d0-47b9-aa2b-993d1474e767 file = sled-storage/src/dataset.rs:388
3492025-02-13T05:05:46.912ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3502025-02-13T05:05:46.912ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3512025-02-13T05:05:46.912ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3522025-02-13T05:05:46.912ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3532025-02-13T05:05:46.912ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3542025-02-13T05:05:46.922ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3552025-02-13T05:05:46.922ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool)
3562025-02-13T05:05:47.021ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }
3572025-02-13T05:05:47.029ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:239
3582025-02-13T05:05:47.029ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:241
3592025-02-13T05:05:47.029ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:245
3602025-02-13T05:05:47.029ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:247
3612025-02-13T05:05:47.029ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3622025-02-13T05:05:47.030ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3632025-02-13T05:05:47.030ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3642025-02-13T05:05:47.064ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3652025-02-13T05:05:47.073ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone file = sled-storage/src/dataset.rs:302
3662025-02-13T05:05:47.156ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
3672025-02-13T05:05:47.211ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a file = sled-storage/src/dataset.rs:388
3682025-02-13T05:05:47.219ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3692025-02-13T05:05:47.219ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3702025-02-13T05:05:47.219ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3712025-02-13T05:05:47.219ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3722025-02-13T05:05:47.219ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3732025-02-13T05:05:47.229ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3742025-02-13T05:05:47.229ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool)
3752025-02-13T05:05:47.326ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }
3762025-02-13T05:05:47.333ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:239
3772025-02-13T05:05:47.333ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:241
3782025-02-13T05:05:47.333ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:245
3792025-02-13T05:05:47.333ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:247
3802025-02-13T05:05:47.333ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3812025-02-13T05:05:47.333ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3822025-02-13T05:05:47.333ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3832025-02-13T05:05:47.369ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3842025-02-13T05:05:47.377ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/zone file = sled-storage/src/dataset.rs:302
3852025-02-13T05:05:47.522ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_888b0ace-3028-4fa4-a464-87268f90ca4e file = sled-storage/src/dataset.rs:388
3862025-02-13T05:05:47.530ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3872025-02-13T05:05:47.530ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3882025-02-13T05:05:47.530ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3892025-02-13T05:05:47.530ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3902025-02-13T05:05:47.530ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3912025-02-13T05:05:47.540ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
3922025-02-13T05:05:47.540ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool)
3932025-02-13T05:05:47.637ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }
3942025-02-13T05:05:47.645ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:239
3952025-02-13T05:05:47.645ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:241
3962025-02-13T05:05:47.645ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:245
3972025-02-13T05:05:47.645ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:247
3982025-02-13T05:05:47.645ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3992025-02-13T05:05:47.645ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4002025-02-13T05:05:47.645ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
4012025-02-13T05:05:47.685ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4022025-02-13T05:05:47.694ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone file = sled-storage/src/dataset.rs:302
4032025-02-13T05:05:47.828ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f file = sled-storage/src/dataset.rs:388
4042025-02-13T05:05:47.836ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4052025-02-13T05:05:47.836ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4062025-02-13T05:05:47.836ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4072025-02-13T05:05:47.836ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4082025-02-13T05:05:47.836ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4092025-02-13T05:05:47.845ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4102025-02-13T05:05:47.845ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool)
4112025-02-13T05:05:47.935ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }
4122025-02-13T05:05:47.943ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:239
4132025-02-13T05:05:47.943ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:241
4142025-02-13T05:05:47.943ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:245
4152025-02-13T05:05:47.943ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:247
4162025-02-13T05:05:47.943ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4172025-02-13T05:05:47.943ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4182025-02-13T05:05:47.943ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
4192025-02-13T05:05:47.979ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4202025-02-13T05:05:47.986ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone file = sled-storage/src/dataset.rs:302
4212025-02-13T05:05:48.132ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_260eeacc-a97f-47c0-b604-100bb7d05b36 file = sled-storage/src/dataset.rs:388
4222025-02-13T05:05:48.142ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4232025-02-13T05:05:48.142ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4242025-02-13T05:05:48.142ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4252025-02-13T05:05:48.142ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(2), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: db71a81c-96cf-41f7-8c27-fbb635fcb5d5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: af97f30c-e09d-4c57-b382-5397ac5562c4 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4262025-02-13T05:05:48.142ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4272025-02-13T05:05:48.142ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4282025-02-13T05:05:48.142ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4292025-02-13T05:05:48.142ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:899 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4302025-02-13T05:05:48.142ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
4312025-02-13T05:05:48.142ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:925 generation = Generation(2) sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4322025-02-13T05:05:48.142ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:930 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4332025-02-13T05:05:48.142ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:935 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4342025-02-13T05:05:48.142ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:123 gen = Generation(2) sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4352025-02-13T05:05:48.142ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:941 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4362025-02-13T05:05:48.142ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:799 gen = Generation(2)
4372025-02-13T05:05:48.142ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:946 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4382025-02-13T05:05:48.142ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3493942 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45474 req_id = 420039fc-64dd-4b65-84f7-a1e1a3b17f7d response_code = 200 uri = /omicron-physical-disks
4392025-02-13T05:05:48.158ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4402025-02-13T05:05:48.174ZINFOSledAgent (RSS): attempting to set up sled's datasets: DatasetsConfig { generation: Generation(1), datasets: {0188bb64-bbf2-485e-9cc9-c9f8bf711fb1 (dataset): DatasetConfig { id: 0188bb64-bbf2-485e-9cc9-c9f8bf711fb1 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0a5739e7-e4bf-4fa0-8130-ab1714cc754b (dataset): DatasetConfig { id: 0a5739e7-e4bf-4fa0-8130-ab1714cc754b (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 0acd866c-d540-4bf5-9a4d-f3e70615feda (dataset): DatasetConfig { id: 0acd866c-d540-4bf5-9a4d-f3e70615feda (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0c7f8cec-d55b-43f1-862d-c06413aa7cde (dataset): DatasetConfig { id: 0c7f8cec-d55b-43f1-862d-c06413aa7cde (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 11b9f182-1734-49b9-ac28-f0ec6dd000a0 (dataset): DatasetConfig { id: 11b9f182-1734-49b9-ac28-f0ec6dd000a0 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 13e3e184-f342-4527-80b1-504bae0e323e (dataset): DatasetConfig { id: 13e3e184-f342-4527-80b1-504bae0e323e (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 147d737c-ed08-4fdd-8a9c-781b5ec0eca8 (dataset): DatasetConfig { id: 147d737c-ed08-4fdd-8a9c-781b5ec0eca8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 16992be2-15bd-4c23-b9ab-6e92c8f28660 (dataset): DatasetConfig { id: 16992be2-15bd-4c23-b9ab-6e92c8f28660 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 17231fa3-589b-499d-954c-6aff6dc115a0 (dataset): DatasetConfig { id: 17231fa3-589b-499d-954c-6aff6dc115a0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1847f372-9a80-46fe-b30b-f770bd5cfc84 (dataset): DatasetConfig { id: 1847f372-9a80-46fe-b30b-f770bd5cfc84 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1a6723cc-51bf-4ac4-9ecb-404d25576527 (dataset): DatasetConfig { id: 1a6723cc-51bf-4ac4-9ecb-404d25576527 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1ddbbb5a-6e27-49cb-954c-42fa7eaa843e (dataset): DatasetConfig { id: 1ddbbb5a-6e27-49cb-954c-42fa7eaa843e (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1f215a7f-895e-4b5c-afee-46031a63b958 (dataset): DatasetConfig { id: 1f215a7f-895e-4b5c-afee-46031a63b958 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 267a40ad-4458-4012-825b-ef3aa1b9bb1f (dataset): DatasetConfig { id: 267a40ad-4458-4012-825b-ef3aa1b9bb1f (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 26f05ee1-2375-4e1d-bb02-028718e480b6 (dataset): DatasetConfig { id: 26f05ee1-2375-4e1d-bb02-028718e480b6 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2d5e21d2-3731-4822-b283-7675b007e431 (dataset): DatasetConfig { id: 2d5e21d2-3731-4822-b283-7675b007e431 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3aaef75c-45d2-4339-8578-5bcb0766ae58 (dataset): DatasetConfig { id: 3aaef75c-45d2-4339-8578-5bcb0766ae58 (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4537d0f7-0dc9-4bdb-a904-72bcb91f9516 (dataset): DatasetConfig { id: 4537d0f7-0dc9-4bdb-a904-72bcb91f9516 (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 46eb6422-0bb3-4e09-a8cb-7445ada4aaff (dataset): DatasetConfig { id: 46eb6422-0bb3-4e09-a8cb-7445ada4aaff (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 478ec97c-bd11-4728-bba9-ff465d9b827b (dataset): DatasetConfig { id: 478ec97c-bd11-4728-bba9-ff465d9b827b (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 47f97a1f-7cc3-490e-8fda-081438f4b490 (dataset): DatasetConfig { id: 47f97a1f-7cc3-490e-8fda-081438f4b490 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5242efda-2cbe-4861-98d7-67f10ec4b92e (dataset): DatasetConfig { id: 5242efda-2cbe-4861-98d7-67f10ec4b92e (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 553a2747-53a5-4959-8140-f53db642d567 (dataset): DatasetConfig { id: 553a2747-53a5-4959-8140-f53db642d567 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 55a64bce-4612-434d-a32e-af288a020e3d (dataset): DatasetConfig { id: 55a64bce-4612-434d-a32e-af288a020e3d (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5663adec-d4f9-4781-8e63-b921138c0fb5 (dataset): DatasetConfig { id: 5663adec-d4f9-4781-8e63-b921138c0fb5 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 59b15b9f-f626-44c1-bba3-dcdac3c14dc2 (dataset): DatasetConfig { id: 59b15b9f-f626-44c1-bba3-dcdac3c14dc2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 59d4f8be-427d-4b78-b46a-bc4b253710a7 (dataset): DatasetConfig { id: 59d4f8be-427d-4b78-b46a-bc4b253710a7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5c693edc-fa99-4cce-a420-4b0f03dbdffa (dataset): DatasetConfig { id: 5c693edc-fa99-4cce-a420-4b0f03dbdffa (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 5d068120-a540-4997-9165-39da2817fc5b (dataset): DatasetConfig { id: 5d068120-a540-4997-9165-39da2817fc5b (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6eb41861-e137-4354-a750-d6a55064746c (dataset): DatasetConfig { id: 6eb41861-e137-4354-a750-d6a55064746c (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 70dcb206-50a0-4256-b45a-bfcd3405c5f2 (dataset): DatasetConfig { id: 70dcb206-50a0-4256-b45a-bfcd3405c5f2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 73030674-d021-4e4b-a7eb-dd06d9c231bb (dataset): DatasetConfig { id: 73030674-d021-4e4b-a7eb-dd06d9c231bb (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 73e0fb83-7ac4-480a-9acc-79b9b28e6605 (dataset): DatasetConfig { id: 73e0fb83-7ac4-480a-9acc-79b9b28e6605 (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 744172c4-7ca1-4272-b7cd-4f4edf29ee8d (dataset): DatasetConfig { id: 744172c4-7ca1-4272-b7cd-4f4edf29ee8d (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 75c7e109-afd7-4bd8-b9a8-36be14fa0af1 (dataset): DatasetConfig { id: 75c7e109-afd7-4bd8-b9a8-36be14fa0af1 (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 76ed5802-bbeb-402d-9c77-899c8e36feba (dataset): DatasetConfig { id: 76ed5802-bbeb-402d-9c77-899c8e36feba (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7db02e78-eb8b-446e-a061-dc7952083b90 (dataset): DatasetConfig { id: 7db02e78-eb8b-446e-a061-dc7952083b90 (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 81842bba-4f54-4b3e-8128-6a29b19d6a3a (dataset): DatasetConfig { id: 81842bba-4f54-4b3e-8128-6a29b19d6a3a (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 89358629-ac71-4190-9e3a-795304c698cb (dataset): DatasetConfig { id: 89358629-ac71-4190-9e3a-795304c698cb (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 89897151-46ea-4965-8ae4-a23b79c95dc0 (dataset): DatasetConfig { id: 89897151-46ea-4965-8ae4-a23b79c95dc0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 8ca5f809-772b-48c8-9159-92bada0c0711 (dataset): DatasetConfig { id: 8ca5f809-772b-48c8-9159-92bada0c0711 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8e7a8fbc-ff70-43f4-a7c0-8a1c3511fda7 (dataset): DatasetConfig { id: 8e7a8fbc-ff70-43f4-a7c0-8a1c3511fda7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 92c3bdc6-4f03-4376-b77e-24b39ffbc0e7 (dataset): DatasetConfig { id: 92c3bdc6-4f03-4376-b77e-24b39ffbc0e7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 94301803-0974-4825-ae9f-8def440cf586 (dataset): DatasetConfig { id: 94301803-0974-4825-ae9f-8def440cf586 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 99e2be90-b931-4171-a67c-b76adcf022bb (dataset): DatasetConfig { id: 99e2be90-b931-4171-a67c-b76adcf022bb (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a00e8f5e-7b14-4c88-8bc6-fa27eae95c0c (dataset): DatasetConfig { id: a00e8f5e-7b14-4c88-8bc6-fa27eae95c0c (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, a0b324d3-0dd5-4a91-b784-01067bd01058 (dataset): DatasetConfig { id: a0b324d3-0dd5-4a91-b784-01067bd01058 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a5d6f2e7-6df6-40a0-900d-538532983097 (dataset): DatasetConfig { id: a5d6f2e7-6df6-40a0-900d-538532983097 (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a68776c1-029a-46ff-948c-efc3f0ba5195 (dataset): DatasetConfig { id: a68776c1-029a-46ff-948c-efc3f0ba5195 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b337abb6-0acd-4fd4-b627-73b4db606dee (dataset): DatasetConfig { id: b337abb6-0acd-4fd4-b627-73b4db606dee (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b57bb741-8d6f-4b0f-9b2e-407a765179d0 (dataset): DatasetConfig { id: b57bb741-8d6f-4b0f-9b2e-407a765179d0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b6e77c58-5dc8-4faa-826d-fd10203d28db (dataset): DatasetConfig { id: b6e77c58-5dc8-4faa-826d-fd10203d28db (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, be1b93c7-66da-43a8-b57f-1f4444d3c709 (dataset): DatasetConfig { id: be1b93c7-66da-43a8-b57f-1f4444d3c709 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c9bffa10-feee-4f4c-af8a-eea6cbdba471 (dataset): DatasetConfig { id: c9bffa10-feee-4f4c-af8a-eea6cbdba471 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, ca5d12ba-23c9-4f97-a750-b7c37c8ff4ff (dataset): DatasetConfig { id: ca5d12ba-23c9-4f97-a750-b7c37c8ff4ff (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dba904d4-f7f1-448f-8797-b0e4d61fe1ef (dataset): DatasetConfig { id: dba904d4-f7f1-448f-8797-b0e4d61fe1ef (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dd1f088c-e134-4e58-bad1-ef4786cdd31e (dataset): DatasetConfig { id: dd1f088c-e134-4e58-bad1-ef4786cdd31e (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e240503c-c02a-42de-bdae-11679376fb01 (dataset): DatasetConfig { id: e240503c-c02a-42de-bdae-11679376fb01 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e5cb7018-cfd4-43eb-8fd7-f674a54694c5 (dataset): DatasetConfig { id: e5cb7018-cfd4-43eb-8fd7-f674a54694c5 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, edfec98d-8cf6-46c1-ae2c-dc6ad8391327 (dataset): DatasetConfig { id: edfec98d-8cf6-46c1-ae2c-dc6ad8391327 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, eec14c14-e977-4223-9219-538818fe19bd (dataset): DatasetConfig { id: eec14c14-e977-4223-9219-538818fe19bd (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f1de2f2c-4f12-4dac-be80-d4b6a6e1288f (dataset): DatasetConfig { id: f1de2f2c-4f12-4dac-be80-d4b6a6e1288f (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, f74e5d8b-3509-48a5-a26c-bdc8dc44841f (dataset): DatasetConfig { id: f74e5d8b-3509-48a5-a26c-bdc8dc44841f (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f88a474b-1b90-4f92-b403-f986e8cf9f81 (dataset): DatasetConfig { id: f88a474b-1b90-4f92-b403-f986e8cf9f81 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fa372737-2409-4229-b627-8392d7109be9 (dataset): DatasetConfig { id: fa372737-2409-4229-b627-8392d7109be9 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fbcec9ca-101e-4632-a03e-6a5d0ff849d4 (dataset): DatasetConfig { id: fbcec9ca-101e-4632-a03e-6a5d0ff849d4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }} } file = sled-agent/src/rack_setup/service.rs:460 sled_address = [fd00:1122:3344:101::1]:12345
4412025-02-13T05:05:48.174ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42203
4422025-02-13T05:05:48.174ZINFOSledAgent: datasets ensure file = sled-agent/src/sled_agent.rs:861 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
4432025-02-13T05:05:48.174ZINFOSledAgent (StorageManager): Received DatasetsEnsure { config: DatasetsConfig { generation: Generation(1), datasets: {0188bb64-bbf2-485e-9cc9-c9f8bf711fb1 (dataset): DatasetConfig { id: 0188bb64-bbf2-485e-9cc9-c9f8bf711fb1 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0a5739e7-e4bf-4fa0-8130-ab1714cc754b (dataset): DatasetConfig { id: 0a5739e7-e4bf-4fa0-8130-ab1714cc754b (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 0acd866c-d540-4bf5-9a4d-f3e70615feda (dataset): DatasetConfig { id: 0acd866c-d540-4bf5-9a4d-f3e70615feda (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0c7f8cec-d55b-43f1-862d-c06413aa7cde (dataset): DatasetConfig { id: 0c7f8cec-d55b-43f1-862d-c06413aa7cde (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 11b9f182-1734-49b9-ac28-f0ec6dd000a0 (dataset): DatasetConfig { id: 11b9f182-1734-49b9-ac28-f0ec6dd000a0 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 13e3e184-f342-4527-80b1-504bae0e323e (dataset): DatasetConfig { id: 13e3e184-f342-4527-80b1-504bae0e323e (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 147d737c-ed08-4fdd-8a9c-781b5ec0eca8 (dataset): DatasetConfig { id: 147d737c-ed08-4fdd-8a9c-781b5ec0eca8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 16992be2-15bd-4c23-b9ab-6e92c8f28660 (dataset): DatasetConfig { id: 16992be2-15bd-4c23-b9ab-6e92c8f28660 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 17231fa3-589b-499d-954c-6aff6dc115a0 (dataset): DatasetConfig { id: 17231fa3-589b-499d-954c-6aff6dc115a0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1847f372-9a80-46fe-b30b-f770bd5cfc84 (dataset): DatasetConfig { id: 1847f372-9a80-46fe-b30b-f770bd5cfc84 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1a6723cc-51bf-4ac4-9ecb-404d25576527 (dataset): DatasetConfig { id: 1a6723cc-51bf-4ac4-9ecb-404d25576527 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1ddbbb5a-6e27-49cb-954c-42fa7eaa843e (dataset): DatasetConfig { id: 1ddbbb5a-6e27-49cb-954c-42fa7eaa843e (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1f215a7f-895e-4b5c-afee-46031a63b958 (dataset): DatasetConfig { id: 1f215a7f-895e-4b5c-afee-46031a63b958 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 267a40ad-4458-4012-825b-ef3aa1b9bb1f (dataset): DatasetConfig { id: 267a40ad-4458-4012-825b-ef3aa1b9bb1f (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 26f05ee1-2375-4e1d-bb02-028718e480b6 (dataset): DatasetConfig { id: 26f05ee1-2375-4e1d-bb02-028718e480b6 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2d5e21d2-3731-4822-b283-7675b007e431 (dataset): DatasetConfig { id: 2d5e21d2-3731-4822-b283-7675b007e431 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3aaef75c-45d2-4339-8578-5bcb0766ae58 (dataset): DatasetConfig { id: 3aaef75c-45d2-4339-8578-5bcb0766ae58 (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4537d0f7-0dc9-4bdb-a904-72bcb91f9516 (dataset): DatasetConfig { id: 4537d0f7-0dc9-4bdb-a904-72bcb91f9516 (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 46eb6422-0bb3-4e09-a8cb-7445ada4aaff (dataset): DatasetConfig { id: 46eb6422-0bb3-4e09-a8cb-7445ada4aaff (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 478ec97c-bd11-4728-bba9-ff465d9b827b (dataset): DatasetConfig { id: 478ec97c-bd11-4728-bba9-ff465d9b827b (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 47f97a1f-7cc3-490e-8fda-081438f4b490 (dataset): DatasetConfig { id: 47f97a1f-7cc3-490e-8fda-081438f4b490 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5242efda-2cbe-4861-98d7-67f10ec4b92e (dataset): DatasetConfig { id: 5242efda-2cbe-4861-98d7-67f10ec4b92e (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 553a2747-53a5-4959-8140-f53db642d567 (dataset): DatasetConfig { id: 553a2747-53a5-4959-8140-f53db642d567 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 55a64bce-4612-434d-a32e-af288a020e3d (dataset): DatasetConfig { id: 55a64bce-4612-434d-a32e-af288a020e3d (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5663adec-d4f9-4781-8e63-b921138c0fb5 (dataset): DatasetConfig { id: 5663adec-d4f9-4781-8e63-b921138c0fb5 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 59b15b9f-f626-44c1-bba3-dcdac3c14dc2 (dataset): DatasetConfig { id: 59b15b9f-f626-44c1-bba3-dcdac3c14dc2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 59d4f8be-427d-4b78-b46a-bc4b253710a7 (dataset): DatasetConfig { id: 59d4f8be-427d-4b78-b46a-bc4b253710a7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5c693edc-fa99-4cce-a420-4b0f03dbdffa (dataset): DatasetConfig { id: 5c693edc-fa99-4cce-a420-4b0f03dbdffa (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 5d068120-a540-4997-9165-39da2817fc5b (dataset): DatasetConfig { id: 5d068120-a540-4997-9165-39da2817fc5b (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6eb41861-e137-4354-a750-d6a55064746c (dataset): DatasetConfig { id: 6eb41861-e137-4354-a750-d6a55064746c (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 70dcb206-50a0-4256-b45a-bfcd3405c5f2 (dataset): DatasetConfig { id: 70dcb206-50a0-4256-b45a-bfcd3405c5f2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 73030674-d021-4e4b-a7eb-dd06d9c231bb (dataset): DatasetConfig { id: 73030674-d021-4e4b-a7eb-dd06d9c231bb (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 73e0fb83-7ac4-480a-9acc-79b9b28e6605 (dataset): DatasetConfig { id: 73e0fb83-7ac4-480a-9acc-79b9b28e6605 (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 744172c4-7ca1-4272-b7cd-4f4edf29ee8d (dataset): DatasetConfig { id: 744172c4-7ca1-4272-b7cd-4f4edf29ee8d (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 75c7e109-afd7-4bd8-b9a8-36be14fa0af1 (dataset): DatasetConfig { id: 75c7e109-afd7-4bd8-b9a8-36be14fa0af1 (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 76ed5802-bbeb-402d-9c77-899c8e36feba (dataset): DatasetConfig { id: 76ed5802-bbeb-402d-9c77-899c8e36feba (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7db02e78-eb8b-446e-a061-dc7952083b90 (dataset): DatasetConfig { id: 7db02e78-eb8b-446e-a061-dc7952083b90 (dataset), name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 81842bba-4f54-4b3e-8128-6a29b19d6a3a (dataset): DatasetConfig { id: 81842bba-4f54-4b3e-8128-6a29b19d6a3a (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 89358629-ac71-4190-9e3a-795304c698cb (dataset): DatasetConfig { id: 89358629-ac71-4190-9e3a-795304c698cb (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 89897151-46ea-4965-8ae4-a23b79c95dc0 (dataset): DatasetConfig { id: 89897151-46ea-4965-8ae4-a23b79c95dc0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 8ca5f809-772b-48c8-9159-92bada0c0711 (dataset): DatasetConfig { id: 8ca5f809-772b-48c8-9159-92bada0c0711 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8e7a8fbc-ff70-43f4-a7c0-8a1c3511fda7 (dataset): DatasetConfig { id: 8e7a8fbc-ff70-43f4-a7c0-8a1c3511fda7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 92c3bdc6-4f03-4376-b77e-24b39ffbc0e7 (dataset): DatasetConfig { id: 92c3bdc6-4f03-4376-b77e-24b39ffbc0e7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 94301803-0974-4825-ae9f-8def440cf586 (dataset): DatasetConfig { id: 94301803-0974-4825-ae9f-8def440cf586 (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 99e2be90-b931-4171-a67c-b76adcf022bb (dataset): DatasetConfig { id: 99e2be90-b931-4171-a67c-b76adcf022bb (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a00e8f5e-7b14-4c88-8bc6-fa27eae95c0c (dataset): DatasetConfig { id: a00e8f5e-7b14-4c88-8bc6-fa27eae95c0c (dataset), name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, a0b324d3-0dd5-4a91-b784-01067bd01058 (dataset): DatasetConfig { id: a0b324d3-0dd5-4a91-b784-01067bd01058 (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a5d6f2e7-6df6-40a0-900d-538532983097 (dataset): DatasetConfig { id: a5d6f2e7-6df6-40a0-900d-538532983097 (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a68776c1-029a-46ff-948c-efc3f0ba5195 (dataset): DatasetConfig { id: a68776c1-029a-46ff-948c-efc3f0ba5195 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b337abb6-0acd-4fd4-b627-73b4db606dee (dataset): DatasetConfig { id: b337abb6-0acd-4fd4-b627-73b4db606dee (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b57bb741-8d6f-4b0f-9b2e-407a765179d0 (dataset): DatasetConfig { id: b57bb741-8d6f-4b0f-9b2e-407a765179d0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b6e77c58-5dc8-4faa-826d-fd10203d28db (dataset): DatasetConfig { id: b6e77c58-5dc8-4faa-826d-fd10203d28db (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, be1b93c7-66da-43a8-b57f-1f4444d3c709 (dataset): DatasetConfig { id: be1b93c7-66da-43a8-b57f-1f4444d3c709 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: TransientZone { name: "oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c9bffa10-feee-4f4c-af8a-eea6cbdba471 (dataset): DatasetConfig { id: c9bffa10-feee-4f4c-af8a-eea6cbdba471 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, ca5d12ba-23c9-4f97-a750-b7c37c8ff4ff (dataset): DatasetConfig { id: ca5d12ba-23c9-4f97-a750-b7c37c8ff4ff (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dba904d4-f7f1-448f-8797-b0e4d61fe1ef (dataset): DatasetConfig { id: dba904d4-f7f1-448f-8797-b0e4d61fe1ef (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dd1f088c-e134-4e58-bad1-ef4786cdd31e (dataset): DatasetConfig { id: dd1f088c-e134-4e58-bad1-ef4786cdd31e (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e240503c-c02a-42de-bdae-11679376fb01 (dataset): DatasetConfig { id: e240503c-c02a-42de-bdae-11679376fb01 (dataset), name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e5cb7018-cfd4-43eb-8fd7-f674a54694c5 (dataset): DatasetConfig { id: e5cb7018-cfd4-43eb-8fd7-f674a54694c5 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, edfec98d-8cf6-46c1-ae2c-dc6ad8391327 (dataset): DatasetConfig { id: edfec98d-8cf6-46c1-ae2c-dc6ad8391327 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, eec14c14-e977-4223-9219-538818fe19bd (dataset): DatasetConfig { id: eec14c14-e977-4223-9219-538818fe19bd (dataset), name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f1de2f2c-4f12-4dac-be80-d4b6a6e1288f (dataset): DatasetConfig { id: f1de2f2c-4f12-4dac-be80-d4b6a6e1288f (dataset), name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, f74e5d8b-3509-48a5-a26c-bdc8dc44841f (dataset): DatasetConfig { id: f74e5d8b-3509-48a5-a26c-bdc8dc44841f (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f88a474b-1b90-4f92-b403-f986e8cf9f81 (dataset): DatasetConfig { id: f88a474b-1b90-4f92-b403-f986e8cf9f81 (dataset), name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fa372737-2409-4229-b627-8392d7109be9 (dataset): DatasetConfig { id: fa372737-2409-4229-b627-8392d7109be9 (dataset), name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fbcec9ca-101e-4632-a03e-6a5d0ff849d4 (dataset): DatasetConfig { id: fbcec9ca-101e-4632-a03e-6a5d0ff849d4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }} }, tx: ... } file = sled-storage/src/manager.rs:607
4442025-02-13T05:05:48.174ZINFOSledAgent (StorageManager): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-datasets.json file = common/src/ledger.rs:188 request = datasets_ensure
4452025-02-13T05:05:48.174ZINFOSledAgent (StorageManager): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-datasets.json file = common/src/ledger.rs:188 request = datasets_ensure
4462025-02-13T05:05:48.174ZINFOSledAgent (StorageManager): No previously-stored 'requested datasets', creating new ledger file = sled-storage/src/manager.rs:909 request = datasets_ensure
4472025-02-13T05:05:48.191ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4482025-02-13T05:05:48.191ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4492025-02-13T05:05:48.191ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone" } file = sled-storage/src/manager.rs:1431
4502025-02-13T05:05:48.218ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4512025-02-13T05:05:48.219ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4522025-02-13T05:05:48.219ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/debug"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/debug" } file = sled-storage/src/manager.rs:1431
4532025-02-13T05:05:48.245ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4542025-02-13T05:05:48.245ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4552025-02-13T05:05:48.245ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crucible" } file = sled-storage/src/manager.rs:1431
4562025-02-13T05:05:48.290ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4572025-02-13T05:05:48.290ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4582025-02-13T05:05:48.290ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_92d599bb-7067-4926-9301-d5f5f983f111/crucible" } file = sled-storage/src/manager.rs:1431
4592025-02-13T05:05:48.338ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4602025-02-13T05:05:48.338ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4612025-02-13T05:05:48.338ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4" } file = sled-storage/src/manager.rs:1431
4622025-02-13T05:05:48.394ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4632025-02-13T05:05:48.394ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4642025-02-13T05:05:48.394ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/zone/oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a"), full_name: "oxp_15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/zone/oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a" } file = sled-storage/src/manager.rs:1431
4652025-02-13T05:05:48.453ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4662025-02-13T05:05:48.453ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4672025-02-13T05:05:48.453ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_9488b08f-08d1-4217-90af-27a0f69b51f7/crucible" } file = sled-storage/src/manager.rs:1431
4682025-02-13T05:05:48.499ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4692025-02-13T05:05:48.499ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4702025-02-13T05:05:48.499ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/external_dns" } file = sled-storage/src/manager.rs:1431
4712025-02-13T05:05:48.545ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4722025-02-13T05:05:48.545ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4732025-02-13T05:05:48.545ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/zone"), full_name: "oxp_9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/zone" } file = sled-storage/src/manager.rs:1431
4742025-02-13T05:05:48.571ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4752025-02-13T05:05:48.571ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4762025-02-13T05:05:48.571ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/zone/oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d"), full_name: "oxp_9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/zone/oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d" } file = sled-storage/src/manager.rs:1431
4772025-02-13T05:05:48.630ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4782025-02-13T05:05:48.630ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4792025-02-13T05:05:48.630ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/external_dns" } file = sled-storage/src/manager.rs:1431
4802025-02-13T05:05:48.678ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4812025-02-13T05:05:48.678ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4822025-02-13T05:05:48.678ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone" } file = sled-storage/src/manager.rs:1431
4832025-02-13T05:05:48.706ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4842025-02-13T05:05:48.706ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4852025-02-13T05:05:48.706ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e" } file = sled-storage/src/manager.rs:1431
4862025-02-13T05:05:48.766ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4872025-02-13T05:05:48.766ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4882025-02-13T05:05:48.766ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/debug"), full_name: "oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/debug" } file = sled-storage/src/manager.rs:1431
4892025-02-13T05:05:48.794ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4902025-02-13T05:05:48.794ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4912025-02-13T05:05:48.794ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258" } file = sled-storage/src/manager.rs:1431
4922025-02-13T05:05:48.852ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4932025-02-13T05:05:48.852ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4942025-02-13T05:05:48.852ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone/oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone/oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3" } file = sled-storage/src/manager.rs:1431
4952025-02-13T05:05:48.909ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4962025-02-13T05:05:48.909ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4972025-02-13T05:05:48.909ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/92d599bb-7067-4926-9301-d5f5f983f111/crypt/zone/oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86"), full_name: "oxp_92d599bb-7067-4926-9301-d5f5f983f111/crypt/zone/oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86" } file = sled-storage/src/manager.rs:1431
4982025-02-13T05:05:48.965ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4992025-02-13T05:05:48.965ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5002025-02-13T05:05:48.965ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/zone"), full_name: "oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/zone" } file = sled-storage/src/manager.rs:1431
5012025-02-13T05:05:48.997ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5022025-02-13T05:05:48.997ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5032025-02-13T05:05:48.997ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228" } file = sled-storage/src/manager.rs:1431
5042025-02-13T05:05:49.053ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5052025-02-13T05:05:49.053ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5062025-02-13T05:05:49.053ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/zone/oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a"), full_name: "oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/zone/oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a" } file = sled-storage/src/manager.rs:1431
5072025-02-13T05:05:49.113ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5082025-02-13T05:05:49.114ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5092025-02-13T05:05:49.114ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
5102025-02-13T05:05:49.160ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
5112025-02-13T05:05:49.160ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5122025-02-13T05:05:49.160ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5132025-02-13T05:05:49.160ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/zone/oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6"), full_name: "oxp_15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/zone/oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6" } file = sled-storage/src/manager.rs:1431
5142025-02-13T05:05:49.220ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5152025-02-13T05:05:49.220ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5162025-02-13T05:05:49.220ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d" } file = sled-storage/src/manager.rs:1431
5172025-02-13T05:05:49.282ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5182025-02-13T05:05:49.282ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5192025-02-13T05:05:49.282ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone" } file = sled-storage/src/manager.rs:1431
5202025-02-13T05:05:49.312ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5212025-02-13T05:05:49.312ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5222025-02-13T05:05:49.312ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55" } file = sled-storage/src/manager.rs:1431
5232025-02-13T05:05:49.374ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5242025-02-13T05:05:49.374ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5252025-02-13T05:05:49.374ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/debug"), full_name: "oxp_15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/debug" } file = sled-storage/src/manager.rs:1431
5262025-02-13T05:05:49.404ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5272025-02-13T05:05:49.404ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5282025-02-13T05:05:49.404ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/zone"), full_name: "oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/zone" } file = sled-storage/src/manager.rs:1431
5292025-02-13T05:05:49.433ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5302025-02-13T05:05:49.433ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5312025-02-13T05:05:49.433ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/debug"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/debug" } file = sled-storage/src/manager.rs:1431
5322025-02-13T05:05:49.462ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5332025-02-13T05:05:49.462ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5342025-02-13T05:05:49.462ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f" } file = sled-storage/src/manager.rs:1431
5352025-02-13T05:05:49.519ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5362025-02-13T05:05:49.519ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5372025-02-13T05:05:49.519ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crucible" } file = sled-storage/src/manager.rs:1431
5382025-02-13T05:05:49.567ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5392025-02-13T05:05:49.567ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5402025-02-13T05:05:49.567ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone/oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone/oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb" } file = sled-storage/src/manager.rs:1431
5412025-02-13T05:05:49.628ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5422025-02-13T05:05:49.628ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5432025-02-13T05:05:49.628ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8" } file = sled-storage/src/manager.rs:1431
5442025-02-13T05:05:49.686ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5452025-02-13T05:05:49.686ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5462025-02-13T05:05:49.686ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/zone/oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e"), full_name: "oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/zone/oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e" } file = sled-storage/src/manager.rs:1431
5472025-02-13T05:05:49.749ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5482025-02-13T05:05:49.749ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5492025-02-13T05:05:49.749ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd" } file = sled-storage/src/manager.rs:1431
5502025-02-13T05:05:49.810ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5512025-02-13T05:05:49.810ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5522025-02-13T05:05:49.810ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/92d599bb-7067-4926-9301-d5f5f983f111/crypt/debug"), full_name: "oxp_92d599bb-7067-4926-9301-d5f5f983f111/crypt/debug" } file = sled-storage/src/manager.rs:1431
5532025-02-13T05:05:49.840ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5542025-02-13T05:05:49.840ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5552025-02-13T05:05:49.840ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/zone/oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b"), full_name: "oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/zone/oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b" } file = sled-storage/src/manager.rs:1431
5562025-02-13T05:05:49.901ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5572025-02-13T05:05:49.901ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5582025-02-13T05:05:49.901ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/92d599bb-7067-4926-9301-d5f5f983f111/crypt/zone"), full_name: "oxp_92d599bb-7067-4926-9301-d5f5f983f111/crypt/zone" } file = sled-storage/src/manager.rs:1431
5592025-02-13T05:05:49.930ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5602025-02-13T05:05:49.930ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5612025-02-13T05:05:49.930ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
5622025-02-13T05:05:49.982ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5632025-02-13T05:05:49.982ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5642025-02-13T05:05:49.982ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
5652025-02-13T05:05:50.033ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5662025-02-13T05:05:50.033ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5672025-02-13T05:05:50.033ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/debug"), full_name: "oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/debug" } file = sled-storage/src/manager.rs:1431
5682025-02-13T05:05:50.063ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5692025-02-13T05:05:50.063ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5702025-02-13T05:05:50.063ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109" } file = sled-storage/src/manager.rs:1431
5712025-02-13T05:05:50.125ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5722025-02-13T05:05:50.125ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5732025-02-13T05:05:50.125ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crucible" } file = sled-storage/src/manager.rs:1431
5742025-02-13T05:05:50.163ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
5752025-02-13T05:05:50.174ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5762025-02-13T05:05:50.174ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5772025-02-13T05:05:50.174ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/internal_dns" } file = sled-storage/src/manager.rs:1431
5782025-02-13T05:05:50.225ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5792025-02-13T05:05:50.225ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5802025-02-13T05:05:50.225ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/zone/oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb"), full_name: "oxp_9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/zone/oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb" } file = sled-storage/src/manager.rs:1431
5812025-02-13T05:05:50.287ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5822025-02-13T05:05:50.287ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5832025-02-13T05:05:50.287ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b" } file = sled-storage/src/manager.rs:1431
5842025-02-13T05:05:50.351ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5852025-02-13T05:05:50.351ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5862025-02-13T05:05:50.351ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/debug"), full_name: "oxp_9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/debug" } file = sled-storage/src/manager.rs:1431
5872025-02-13T05:05:50.382ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5882025-02-13T05:05:50.382ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5892025-02-13T05:05:50.382ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone/oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/zone/oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f" } file = sled-storage/src/manager.rs:1431
5902025-02-13T05:05:50.442ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5912025-02-13T05:05:50.442ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5922025-02-13T05:05:50.442ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/zone"), full_name: "oxp_15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/zone" } file = sled-storage/src/manager.rs:1431
5932025-02-13T05:05:50.474ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5942025-02-13T05:05:50.474ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5952025-02-13T05:05:50.474ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/clickhouse" } file = sled-storage/src/manager.rs:1431
5962025-02-13T05:05:50.526ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5972025-02-13T05:05:50.526ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5982025-02-13T05:05:50.526ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crucible" } file = sled-storage/src/manager.rs:1431
5992025-02-13T05:05:50.577ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6002025-02-13T05:05:50.577ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6012025-02-13T05:05:50.577ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crucible" } file = sled-storage/src/manager.rs:1431
6022025-02-13T05:05:50.627ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6032025-02-13T05:05:50.627ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6042025-02-13T05:05:50.627ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168" } file = sled-storage/src/manager.rs:1431
6052025-02-13T05:05:50.686ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6062025-02-13T05:05:50.686ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6072025-02-13T05:05:50.686ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/zone/oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37" } file = sled-storage/src/manager.rs:1431
6082025-02-13T05:05:50.748ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6092025-02-13T05:05:50.748ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6102025-02-13T05:05:50.748ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/debug"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/debug" } file = sled-storage/src/manager.rs:1431
6112025-02-13T05:05:50.778ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6122025-02-13T05:05:50.778ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6132025-02-13T05:05:50.778ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/internal_dns" } file = sled-storage/src/manager.rs:1431
6142025-02-13T05:05:50.827ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6152025-02-13T05:05:50.828ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6162025-02-13T05:05:50.828ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
6172025-02-13T05:05:50.879ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6182025-02-13T05:05:50.879ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6192025-02-13T05:05:50.879ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909" } file = sled-storage/src/manager.rs:1431
6202025-02-13T05:05:50.941ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6212025-02-13T05:05:50.941ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6222025-02-13T05:05:50.941ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crucible" } file = sled-storage/src/manager.rs:1431
6232025-02-13T05:05:50.991ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6242025-02-13T05:05:50.991ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6252025-02-13T05:05:50.991ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8" } file = sled-storage/src/manager.rs:1431
6262025-02-13T05:05:51.052ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6272025-02-13T05:05:51.052ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6282025-02-13T05:05:51.052ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/zone/oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496" } file = sled-storage/src/manager.rs:1431
6292025-02-13T05:05:51.113ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6302025-02-13T05:05:51.113ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6312025-02-13T05:05:51.113ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
6322025-02-13T05:05:51.165ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6332025-02-13T05:05:51.165ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6342025-02-13T05:05:51.165ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/debug"), full_name: "oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/debug" } file = sled-storage/src/manager.rs:1431
6352025-02-13T05:05:51.168ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
6362025-02-13T05:05:51.195ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6372025-02-13T05:05:51.195ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6382025-02-13T05:05:51.195ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone" } file = sled-storage/src/manager.rs:1431
6392025-02-13T05:05:51.226ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6402025-02-13T05:05:51.226ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6412025-02-13T05:05:51.226ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd"), full_name: "oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/zone/oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd" } file = sled-storage/src/manager.rs:1431
6422025-02-13T05:05:51.286ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6432025-02-13T05:05:51.286ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6442025-02-13T05:05:51.286ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/internal_dns" } file = sled-storage/src/manager.rs:1431
6452025-02-13T05:05:51.337ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6462025-02-13T05:05:51.337ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6472025-02-13T05:05:51.337ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_15d1957c-72d0-47b9-aa2b-993d1474e767/crucible" } file = sled-storage/src/manager.rs:1431
6482025-02-13T05:05:51.387ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.omicron-datasets.json.tmp file = common/src/ledger.rs:199 request = datasets_ensure
6492025-02-13T05:05:51.387ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.omicron-datasets.json.tmp file = common/src/ledger.rs:199 request = datasets_ensure
6502025-02-13T05:05:51.387ZINFOSledAgent: datasets ensure: Updated storage file = sled-agent/src/sled_agent.rs:863 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
6512025-02-13T05:05:51.387ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3213338 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42203 req_id = 891bc6b3-2e96-4c85-bf0e-0629a632c21d response_code = 200 uri = /datasets
6522025-02-13T05:05:51.420ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
6532025-02-13T05:05:51.420ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37533
6542025-02-13T05:05:51.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
6552025-02-13T05:05:51.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
6562025-02-13T05:05:51.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
6572025-02-13T05:05:51.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6582025-02-13T05:05:51.438ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
6592025-02-13T05:05:51.438ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6602025-02-13T05:05:51.447ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6612025-02-13T05:05:51.447ZINFOSledAgent (ServiceManager): No ledger in /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:188
6622025-02-13T05:05:51.447ZINFOSledAgent (ServiceManager): No ledger in /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:188
6632025-02-13T05:05:51.447ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 3 num_zones_to_be_removed = 0 num_zones_updated = 0
6642025-02-13T05:05:51.447ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6652025-02-13T05:05:51.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6662025-02-13T05:05:51.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6672025-02-13T05:05:51.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6682025-02-13T05:05:51.544ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258 file = illumos-utils/src/zone.rs:325
6692025-02-13T05:05:51.548ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd file = illumos-utils/src/zone.rs:325
6702025-02-13T05:05:51.552ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb file = illumos-utils/src/zone.rs:325
6712025-02-13T05:05:51.565ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258 file = illumos-utils/src/zone.rs:363
6722025-02-13T05:05:51.570ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd file = illumos-utils/src/zone.rs:363
6732025-02-13T05:05:51.574ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb file = illumos-utils/src/zone.rs:363
6742025-02-13T05:05:52.170ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/probe_manager.rs:238 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
6752025-02-13T05:05:53.170ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6762025-02-13T05:05:53.170ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
6772025-02-13T05:05:53.170ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
6782025-02-13T05:05:53.656ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6792025-02-13T05:05:53.657ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
6802025-02-13T05:05:53.997ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1880608015253843444 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
6812025-02-13T05:05:53.997ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12669931516114820383 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
6822025-02-13T05:05:54.138ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6832025-02-13T05:05:54.146ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 53.981609ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
6842025-02-13T05:05:54.146ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
6852025-02-13T05:05:54.151ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
6862025-02-13T05:05:54.151ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
6872025-02-13T05:05:54.211ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 109.548281ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
6882025-02-13T05:05:54.217ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4364
6892025-02-13T05:05:54.217ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4374 gateway = fd00:1122:3344:101::1
6902025-02-13T05:05:54.224ZINFOSledAgent (ServiceManager): Default route already exists file = sled-agent/src/services.rs:4388 gateway = fd00:1122:3344:101::1
6912025-02-13T05:05:54.224ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4453
6922025-02-13T05:05:54.282ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 65.053157ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
6932025-02-13T05:05:54.291ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 71.877063ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
6942025-02-13T05:05:54.328ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 176.401122ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
6952025-02-13T05:05:54.459ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 82.76083ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
6962025-02-13T05:05:54.549ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 73.387208ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
6972025-02-13T05:05:54.578ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4494
6982025-02-13T05:05:54.578ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4521
6992025-02-13T05:05:54.587ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 304.316854ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
7002025-02-13T05:05:54.624ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 293.74096ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7012025-02-13T05:05:54.668ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 190.816336ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7022025-02-13T05:05:54.756ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4534
7032025-02-13T05:05:54.756ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4405
7042025-02-13T05:05:54.827ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4447
7052025-02-13T05:05:54.827ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4501 rack_subnet = fd00:1122:3344::
7062025-02-13T05:05:54.851ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4512
7072025-02-13T05:05:54.851ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4585
7082025-02-13T05:05:54.872ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 592.194081ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7092025-02-13T05:05:54.901ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 999.027415ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
7102025-02-13T05:05:54.930ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 507.64143ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7112025-02-13T05:05:54.935ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4617
7122025-02-13T05:05:54.935ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4623
7132025-02-13T05:05:55.055ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4664
7142025-02-13T05:05:55.056ZINFOSledAgent (ServiceManager): configuring tfport service file = sled-agent/src/services.rs:4537
7152025-02-13T05:05:55.153ZINFOSledAgent (ServiceManager): refreshed tfport service with new configuration file = sled-agent/src/services.rs:4570
7162025-02-13T05:05:55.153ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
7172025-02-13T05:05:55.186ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/nexus.rs:530 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
7182025-02-13T05:05:55.187ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:420
7192025-02-13T05:05:55.219ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:717
7202025-02-13T05:05:55.219ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:762
7212025-02-13T05:05:55.220ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:765 uptime = 2
7222025-02-13T05:05:55.220ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: Some(\n None,\n ),\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n tx_eq: None,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:444
7232025-02-13T05:05:55.253ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11041192580377101784 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
7242025-02-13T05:05:55.253ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14879566039012961736 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
7252025-02-13T05:05:55.256ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
7262025-02-13T05:05:55.310ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:607
7272025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
7282025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
7292025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
7302025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
7312025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
7322025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
7332025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
7342025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
7352025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
7362025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
7372025-02-13T05:05:55.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
7382025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
7392025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
7402025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
7412025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
7422025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
7432025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
7442025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
7452025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
7462025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
7472025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
7482025-02-13T05:05:55.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
7492025-02-13T05:05:55.449ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 445.398719ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7502025-02-13T05:05:55.474ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 1.054157097s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7512025-02-13T05:05:55.904ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 1.265304961s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7522025-02-13T05:05:55.913ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 1.078772144s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
7532025-02-13T05:05:56.538ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 851.6428ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7542025-02-13T05:05:57.002ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 720.59229ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
7552025-02-13T05:05:57.179ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 722.737584ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7562025-02-13T05:05:57.398ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 1.032069404s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7572025-02-13T05:05:57.731ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 1.478469271s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
7582025-02-13T05:05:57.911ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 849.840163ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7592025-02-13T05:05:58.439ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 1.082537283s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7602025-02-13T05:05:58.769ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 734.537676ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7612025-02-13T05:05:59.219ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd") failed: Property not found. retry in 1.127997918s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
7622025-02-13T05:05:59.514ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb") failed: Property not found. retry in 673.169923ms file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7632025-02-13T05:05:59.531ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258") failed: Property not found. retry in 1.441052769s file = illumos-utils/src/svc.rs:42 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7642025-02-13T05:06:00.993ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_internal_dns_9f4eb8b3-c464-433c-b912-c7432c124258
7652025-02-13T05:06:00.993ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_internal_dns_cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd
7662025-02-13T05:06:00.993ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_internal_dns_d80d65e7-849e-47a7-8d7e-e69f7cc544bb
7672025-02-13T05:06:00.993ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7682025-02-13T05:06:00.993ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7692025-02-13T05:06:00.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 9573568 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37533 req_id = 2034c5cf-813f-4e85-baa0-f9837b649896 response_code = 204 uri = /omicron-zones
7702025-02-13T05:06:00.997ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:658
7712025-02-13T05:06:02.194ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:658
7722025-02-13T05:06:02.270ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:658
7732025-02-13T05:06:02.345ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:694
7742025-02-13T05:06:02.346ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
7752025-02-13T05:06:02.378ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
7762025-02-13T05:06:02.378ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
7772025-02-13T05:06:02.378ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
7782025-02-13T05:06:02.408ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
7792025-02-13T05:06:02.408ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58256
7802025-02-13T05:06:02.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
7812025-02-13T05:06:02.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
7822025-02-13T05:06:02.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
7832025-02-13T05:06:02.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7842025-02-13T05:06:02.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
7852025-02-13T05:06:02.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7862025-02-13T05:06:02.440ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
7872025-02-13T05:06:02.440ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
7882025-02-13T05:06:02.440ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
7892025-02-13T05:06:02.440ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 1 num_zones_to_be_removed = 0 num_zones_updated = 0
7902025-02-13T05:06:02.440ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
7912025-02-13T05:06:02.452ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
7922025-02-13T05:06:02.484ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
7932025-02-13T05:06:02.485ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
7942025-02-13T05:06:02.485ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
7952025-02-13T05:06:02.515ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 181, 161]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
7962025-02-13T05:06:02.515ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1321 zone_type = boundary_ntp
7972025-02-13T05:06:02.515ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
7982025-02-13T05:06:02.515ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
7992025-02-13T05:06:02.515ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
8002025-02-13T05:06:02.516ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
8012025-02-13T05:06:02.539ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37 file = illumos-utils/src/zone.rs:325
8022025-02-13T05:06:02.557ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37 file = illumos-utils/src/zone.rs:363
8032025-02-13T05:06:03.189ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8042025-02-13T05:06:03.190ZINFOSledAgent (ServiceManager): Booting oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8052025-02-13T05:06:03.319ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 74.239753ms file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8062025-02-13T05:06:03.403ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 53.729694ms file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8072025-02-13T05:06:03.467ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 267.499528ms file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8082025-02-13T05:06:03.748ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 339.989293ms file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8092025-02-13T05:06:04.000ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1880608015253843444 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8102025-02-13T05:06:04.000ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12669931516114820383 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8112025-02-13T05:06:04.097ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 1.199201043s file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8122025-02-13T05:06:05.255ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11041192580377101784 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8132025-02-13T05:06:05.255ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14879566039012961736 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8142025-02-13T05:06:05.307ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 1.308230988s file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8152025-02-13T05:06:06.625ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 1.010733675s file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8162025-02-13T05:06:07.645ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 648.357418ms file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8172025-02-13T05:06:08.303ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37") failed: Property not found. retry in 842.417181ms file = illumos-utils/src/svc.rs:42 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8182025-02-13T05:06:09.165ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37
8192025-02-13T05:06:09.165ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8202025-02-13T05:06:09.165ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8212025-02-13T05:06:09.165ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 6757494 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:58256 req_id = d4c321e5-cf06-4a55-a279-e8b14360dd82 response_code = 204 uri = /omicron-zones
8222025-02-13T05:06:09.168ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:735
8232025-02-13T05:06:09.197ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8242025-02-13T05:06:09.197ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37424
8252025-02-13T05:06:09.207ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3972
8262025-02-13T05:06:09.207ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 9385 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37424 req_id = 83a2f304-35e4-48ca-9711-29b0f3f939e6 response_code = 500 uri = /timesync
8272025-02-13T05:06:09.210ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8282025-02-13T05:06:09.389ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8292025-02-13T05:06:09.389ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46456
8302025-02-13T05:06:10.213ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2339203509661629204 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8312025-02-13T05:06:10.370ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11674776510630608998 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8322025-02-13T05:06:10.996ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 128295734488100015 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8332025-02-13T05:06:13.397ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3972
8342025-02-13T05:06:13.397ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 4008449 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46456 req_id = c59fe759-1709-42df-9dd1-82dd3a06114d response_code = 500 uri = /timesync
8352025-02-13T05:06:13.400ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8362025-02-13T05:06:13.561ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8372025-02-13T05:06:13.561ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55054
8382025-02-13T05:06:13.570ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3972
8392025-02-13T05:06:13.570ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 8828 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55054 req_id = db3205df-9536-4ea4-a0d9-dc665462388f response_code = 500 uri = /timesync
8402025-02-13T05:06:13.573ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8412025-02-13T05:06:13.791ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8422025-02-13T05:06:13.791ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48503
8432025-02-13T05:06:13.799ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_133265e1-ac44-46da-8aab-7104695c8e37': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3972
8442025-02-13T05:06:13.800ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 8567 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48503 req_id = bf8c1ce5-4cd0-4d48-8acd-8f5e01c3382c response_code = 500 uri = /timesync
8452025-02-13T05:06:13.802ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8462025-02-13T05:06:14.002ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1880608015253843444 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8472025-02-13T05:06:14.002ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12669931516114820383 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8482025-02-13T05:06:14.075ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8492025-02-13T05:06:14.075ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61506
8502025-02-13T05:06:15.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1008888 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61506 req_id = cd799ca4-4f7a-4ff6-8442-1581fb7ab3d3 response_code = 200 uri = /timesync
8512025-02-13T05:06:15.087ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8522025-02-13T05:06:15.087ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8532025-02-13T05:06:15.257ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11041192580377101784 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8542025-02-13T05:06:15.257ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14879566039012961736 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
8552025-02-13T05:06:15.386ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8562025-02-13T05:06:15.386ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61189
8572025-02-13T05:06:15.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 8389 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61189 req_id = 1560ee68-5642-47f2-8f36-6380e70e3cb9 response_code = 200 uri = /timesync
8582025-02-13T05:06:15.396ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8592025-02-13T05:06:15.396ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8602025-02-13T05:06:15.697ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8612025-02-13T05:06:15.697ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59952
8622025-02-13T05:06:15.706ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 8584 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59952 req_id = 15cfa365-03e7-4eef-8ebd-27d493f17f2b response_code = 200 uri = /timesync
8632025-02-13T05:06:15.709ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8642025-02-13T05:06:15.709ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8652025-02-13T05:06:15.915ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8662025-02-13T05:06:15.915ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58059
8672025-02-13T05:06:15.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 8347 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58059 req_id = 2fac04ee-d153-40ce-80ba-10f3eb82a608 response_code = 200 uri = /timesync
8682025-02-13T05:06:15.925ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8692025-02-13T05:06:15.925ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8702025-02-13T05:06:16.142ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8712025-02-13T05:06:16.143ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62601
8722025-02-13T05:06:16.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 9501 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62601 req_id = e6d90cc6-a3a6-456f-84ca-32c2c9165814 response_code = 200 uri = /timesync
8732025-02-13T05:06:16.155ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8742025-02-13T05:06:16.155ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8752025-02-13T05:06:16.360ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8762025-02-13T05:06:16.361ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58911
8772025-02-13T05:06:16.381ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 19864 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58911 req_id = d9eab5b1-58e9-48cd-8ec3-ffddd662a0d2 response_code = 200 uri = /timesync
8782025-02-13T05:06:16.384ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2501691571, ip_addr: 149.28.200.179, stratum: 3, ref_time: 1739423175.3717558, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8792025-02-13T05:06:16.384ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1383
8802025-02-13T05:06:16.413ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
8812025-02-13T05:06:16.414ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40959
8822025-02-13T05:06:16.414ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
8832025-02-13T05:06:16.414ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
8842025-02-13T05:06:16.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
8852025-02-13T05:06:16.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8862025-02-13T05:06:16.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
8872025-02-13T05:06:16.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8882025-02-13T05:06:17.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
8892025-02-13T05:06:17.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8902025-02-13T05:06:17.020ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
8912025-02-13T05:06:17.020ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8922025-02-13T05:06:17.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
8932025-02-13T05:06:17.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8942025-02-13T05:06:17.039ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
8952025-02-13T05:06:17.039ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
8962025-02-13T05:06:17.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
8972025-02-13T05:06:17.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
8982025-02-13T05:06:17.056ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
8992025-02-13T05:06:17.057ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
9002025-02-13T05:06:17.057ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
9012025-02-13T05:06:17.057ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 5 num_zones_to_be_removed = 0 num_zones_updated = 0
9022025-02-13T05:06:17.057ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
9032025-02-13T05:06:17.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
9042025-02-13T05:06:17.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
9052025-02-13T05:06:17.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
9062025-02-13T05:06:17.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
9072025-02-13T05:06:17.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
9082025-02-13T05:06:17.150ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168 file = illumos-utils/src/zone.rs:325
9092025-02-13T05:06:17.154ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8 file = illumos-utils/src/zone.rs:325
9102025-02-13T05:06:17.158ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a file = illumos-utils/src/zone.rs:325
9112025-02-13T05:06:17.162ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4 file = illumos-utils/src/zone.rs:325
9122025-02-13T05:06:17.167ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3 file = illumos-utils/src/zone.rs:325
9132025-02-13T05:06:17.171ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168 file = illumos-utils/src/zone.rs:363
9142025-02-13T05:06:17.176ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8 file = illumos-utils/src/zone.rs:363
9152025-02-13T05:06:17.181ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a file = illumos-utils/src/zone.rs:363
9162025-02-13T05:06:17.185ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4 file = illumos-utils/src/zone.rs:363
9172025-02-13T05:06:17.190ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3 file = illumos-utils/src/zone.rs:363
9182025-02-13T05:06:20.252ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='f2470d0d-da0c-4e4f-a170-b6766c01e168'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9192025-02-13T05:06:20.252ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9202025-02-13T05:06:20.323ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='40ba9488-97a0-4c82-8eaa-572a3d05f4a3'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9212025-02-13T05:06:20.324ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9222025-02-13T05:06:20.354ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='4f62b307-7a1b-4e58-b809-072df20f86f4'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9232025-02-13T05:06:20.354ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9242025-02-13T05:06:20.386ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='9b13677c-db2d-419d-8efe-35f1b21ab4b8'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9252025-02-13T05:06:20.386ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9262025-02-13T05:06:20.407ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 38.634391ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9272025-02-13T05:06:20.407ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3bd70f31-983e-48d1-b8d7-426a36f6d46a'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9282025-02-13T05:06:20.407ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9292025-02-13T05:06:20.456ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 58.310583ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9302025-02-13T05:06:20.467ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 63.239861ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9312025-02-13T05:06:20.496ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 29.114016ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9322025-02-13T05:06:20.527ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 285.200081ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9332025-02-13T05:06:20.539ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 104.87585ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9342025-02-13T05:06:20.628ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 86.37052ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9352025-02-13T05:06:20.706ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 38.57793ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9362025-02-13T05:06:21.070ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 352.41798ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
9372025-02-13T05:06:21.244ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 29.373208ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9382025-02-13T05:06:21.403ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 125.266851ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9392025-02-13T05:06:21.431ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 132.901314ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9402025-02-13T05:06:21.506ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 340.940861ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9412025-02-13T05:06:21.571ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 66.873425ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9422025-02-13T05:06:21.626ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 53.804323ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9432025-02-13T05:06:21.659ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 218.635755ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9442025-02-13T05:06:21.678ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 230.876122ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9452025-02-13T05:06:21.692ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 289.52858ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9462025-02-13T05:06:21.704ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 114.748792ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9472025-02-13T05:06:21.832ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 417.130771ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9482025-02-13T05:06:21.866ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 787.523797ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9492025-02-13T05:06:21.890ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 1.113572275s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9502025-02-13T05:06:21.919ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 588.542752ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9512025-02-13T05:06:21.991ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 501.049657ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9522025-02-13T05:06:22.260ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 816.645005ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9532025-02-13T05:06:22.505ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 444.743748ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9542025-02-13T05:06:22.518ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 656.336491ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9552025-02-13T05:06:22.663ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 1.485377991s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9562025-02-13T05:06:22.958ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.028787753s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9572025-02-13T05:06:23.012ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 657.666041ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9582025-02-13T05:06:23.086ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 894.20777ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9592025-02-13T05:06:23.183ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 522.082511ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9602025-02-13T05:06:23.677ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 1.334298348s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9612025-02-13T05:06:23.713ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 597.124496ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9622025-02-13T05:06:23.987ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 1.235970548s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9632025-02-13T05:06:23.996ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.279834713s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9642025-02-13T05:06:24.154ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 1.202550238s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9652025-02-13T05:06:24.318ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 851.124965ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9662025-02-13T05:06:24.658ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58961
9672025-02-13T05:06:24.659ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 75 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58961 req_id = 25fbbd26-4575-45a7-b929-fee231a40d75 response_code = 200 uri = /baseboard
9682025-02-13T05:06:25.018ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 623.560189ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9692025-02-13T05:06:25.176ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 560.137293ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9702025-02-13T05:06:25.228ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 500.386746ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9712025-02-13T05:06:25.281ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 686.506804ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9722025-02-13T05:06:25.361ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 1.333173891s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9732025-02-13T05:06:25.648ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 587.138741ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9742025-02-13T05:06:25.736ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 716.599203ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9752025-02-13T05:06:25.743ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 991.155951ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9762025-02-13T05:06:25.975ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.32586556s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9772025-02-13T05:06:26.242ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 1.437324537s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9782025-02-13T05:06:26.461ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 1.284012176s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9792025-02-13T05:06:26.700ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 701.651715ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9802025-02-13T05:06:26.740ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 1.406436155s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9812025-02-13T05:06:27.307ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 719.840109ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9822025-02-13T05:06:27.408ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 838.049876ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9832025-02-13T05:06:27.685ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 1.142403529s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9842025-02-13T05:06:27.751ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 886.003423ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9852025-02-13T05:06:28.038ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.065083766s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9862025-02-13T05:06:28.151ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 741.349463ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9872025-02-13T05:06:28.252ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 1.075823492s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9882025-02-13T05:06:28.644ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 535.157017ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9892025-02-13T05:06:28.835ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 1.391589399s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9902025-02-13T05:06:28.899ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 1.049329994s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9912025-02-13T05:06:29.111ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.328835166s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9922025-02-13T05:06:29.186ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 1.04152343s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9932025-02-13T05:06:29.335ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 1.102188913s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9942025-02-13T05:06:29.956ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 1.237894946s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
9952025-02-13T05:06:30.233ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 945.678498ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
9962025-02-13T05:06:30.242ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 1.052401137s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
9972025-02-13T05:06:30.443ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 796.159637ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
9982025-02-13T05:06:30.451ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.110967446s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
9992025-02-13T05:06:31.186ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 822.302658ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
10002025-02-13T05:06:31.201ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 1.322034677s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
10012025-02-13T05:06:31.246ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 890.9883ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
10022025-02-13T05:06:31.300ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 1.072654735s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
10032025-02-13T05:06:31.568ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.48764714s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10042025-02-13T05:06:32.016ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 833.861703ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
10052025-02-13T05:06:32.145ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 884.063166ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
10062025-02-13T05:06:32.380ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 518.026824ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
10072025-02-13T05:06:32.531ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 1.037044543s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
10082025-02-13T05:06:32.858ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 543.057297ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
10092025-02-13T05:06:32.906ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 577.249163ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
10102025-02-13T05:06:33.036ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168") failed: Property not found. retry in 1.398761811s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
10112025-02-13T05:06:33.061ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 567.294944ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10122025-02-13T05:06:33.408ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4") failed: Property not found. retry in 1.221231558s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
10132025-02-13T05:06:33.490ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 808.259193ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
10142025-02-13T05:06:33.573ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 920.102759ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
10152025-02-13T05:06:33.636ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 619.147618ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10162025-02-13T05:06:34.264ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.413704088s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10172025-02-13T05:06:34.305ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a") failed: Property not found. retry in 543.36425ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
10182025-02-13T05:06:34.500ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3") failed: Property not found. retry in 919.169064ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
10192025-02-13T05:06:35.685ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 781.331476ms file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10202025-02-13T05:06:36.473ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.093509563s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10212025-02-13T05:06:36.620ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 635.426651ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
10222025-02-13T05:06:37.576ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.391945048s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10232025-02-13T05:06:39.473ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.072077273s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10242025-02-13T05:06:40.556ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.120614944s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10252025-02-13T05:06:41.684ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.061044576s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10262025-02-13T05:06:42.754ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.025312256s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10272025-02-13T05:06:43.786ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8") failed: Property not found. retry in 1.430532727s file = illumos-utils/src/svc.rs:42 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10282025-02-13T05:06:45.234ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_cockroachdb_f2470d0d-da0c-4e4f-a170-b6766c01e168
10292025-02-13T05:06:45.234ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_cockroachdb_9b13677c-db2d-419d-8efe-35f1b21ab4b8
10302025-02-13T05:06:45.234ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_cockroachdb_3bd70f31-983e-48d1-b8d7-426a36f6d46a
10312025-02-13T05:06:45.234ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_cockroachdb_4f62b307-7a1b-4e58-b809-072df20f86f4
10322025-02-13T05:06:45.234ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_cockroachdb_40ba9488-97a0-4c82-8eaa-572a3d05f4a3
10332025-02-13T05:06:45.234ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
10342025-02-13T05:06:45.240ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
10352025-02-13T05:06:45.241ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 28413989 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:40959 req_id = a7b1cabd-04fa-477e-b64b-b5e295498db3 response_code = 204 uri = /omicron-zones
10362025-02-13T05:06:45.273ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48961
10372025-02-13T05:06:45.312ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3815
10382025-02-13T05:06:45.591ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3833
10392025-02-13T05:06:50.737ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3856
10402025-02-13T05:06:50.737ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5478439 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:48961 req_id = e831e5da-20b8-46ef-9274-33d1f082e07b response_code = 204 uri = /cockroachdb
10412025-02-13T05:06:50.772ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
10422025-02-13T05:06:50.772ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53368
10432025-02-13T05:06:50.773ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
10442025-02-13T05:06:50.773ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
10452025-02-13T05:06:50.786ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
10462025-02-13T05:06:50.786ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10472025-02-13T05:06:50.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
10482025-02-13T05:06:50.797ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10492025-02-13T05:06:50.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
10502025-02-13T05:06:50.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10512025-02-13T05:06:50.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
10522025-02-13T05:06:50.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10532025-02-13T05:06:50.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
10542025-02-13T05:06:50.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10552025-02-13T05:06:50.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
10562025-02-13T05:06:50.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
10572025-02-13T05:06:50.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
10582025-02-13T05:06:50.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
10592025-02-13T05:06:50.858ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
10602025-02-13T05:06:50.858ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10612025-02-13T05:06:50.868ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
10622025-02-13T05:06:50.868ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10632025-02-13T05:06:50.878ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
10642025-02-13T05:06:50.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10652025-02-13T05:06:50.889ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10662025-02-13T05:06:50.889ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
10672025-02-13T05:06:50.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10682025-02-13T05:06:50.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
10692025-02-13T05:06:50.908ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10702025-02-13T05:06:50.908ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
10712025-02-13T05:06:50.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10722025-02-13T05:06:50.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
10732025-02-13T05:06:50.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10742025-02-13T05:06:50.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
10752025-02-13T05:06:50.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10762025-02-13T05:06:50.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
10772025-02-13T05:06:50.950ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10782025-02-13T05:06:50.950ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
10792025-02-13T05:06:50.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10802025-02-13T05:06:50.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
10812025-02-13T05:06:50.970ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
10822025-02-13T05:06:50.971ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
10832025-02-13T05:06:50.980ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
10842025-02-13T05:06:50.980ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
10852025-02-13T05:06:50.980ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
10862025-02-13T05:06:50.981ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
10872025-02-13T05:06:50.981ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
10882025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
10892025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
10902025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
10912025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
10922025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
10932025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
10942025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
10952025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
10962025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
10972025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
10982025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
10992025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
11002025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
11012025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
11022025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
11032025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
11042025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
11052025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
11062025-02-13T05:06:51.092ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
11072025-02-13T05:06:51.092ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 320909 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = ac48c1e7-7ec2-4ffc-968e-15b64b8332ef response_code = 503 uri = /omicron-zones
11082025-02-13T05:06:51.093ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "ac48c1e7-7ec2-4ffc-968e-15b64b8332ef", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:50 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "ac48c1e7-7ec2-4ffc-968e-15b64b8332ef" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 59.060521ms sled_address = [fd00:1122:3344:101::1]:12345
11092025-02-13T05:06:51.153ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
11102025-02-13T05:06:51.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11112025-02-13T05:06:51.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
11122025-02-13T05:06:51.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11132025-02-13T05:06:51.164ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11142025-02-13T05:06:51.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11152025-02-13T05:06:51.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11162025-02-13T05:06:51.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11172025-02-13T05:06:51.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11182025-02-13T05:06:51.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11192025-02-13T05:06:51.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11202025-02-13T05:06:51.206ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11212025-02-13T05:06:51.206ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11222025-02-13T05:06:51.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11232025-02-13T05:06:51.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
11242025-02-13T05:06:51.226ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11252025-02-13T05:06:51.226ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
11262025-02-13T05:06:51.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11272025-02-13T05:06:51.236ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11282025-02-13T05:06:51.246ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11292025-02-13T05:06:51.246ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11302025-02-13T05:06:51.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
11312025-02-13T05:06:51.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11322025-02-13T05:06:51.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11332025-02-13T05:06:51.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
11342025-02-13T05:06:51.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11352025-02-13T05:06:51.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
11362025-02-13T05:06:51.284ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11372025-02-13T05:06:51.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
11382025-02-13T05:06:51.294ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11392025-02-13T05:06:51.294ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
11402025-02-13T05:06:51.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11412025-02-13T05:06:51.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
11422025-02-13T05:06:51.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11432025-02-13T05:06:51.316ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
11442025-02-13T05:06:51.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11452025-02-13T05:06:51.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
11462025-02-13T05:06:51.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11472025-02-13T05:06:51.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
11482025-02-13T05:06:51.345ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
11492025-02-13T05:06:51.345ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
11502025-02-13T05:06:51.354ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
11512025-02-13T05:06:51.354ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
11522025-02-13T05:06:51.355ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
11532025-02-13T05:06:51.355ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
11542025-02-13T05:06:51.355ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
11552025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
11562025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
11572025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
11582025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
11592025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
11602025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
11612025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
11622025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
11632025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
11642025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
11652025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
11662025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
11672025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
11682025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
11692025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
11702025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
11712025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
11722025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
11732025-02-13T05:06:51.453ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
11742025-02-13T05:06:51.453ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 301034 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 3df63878-29ae-49ca-89aa-73486f07bbcf response_code = 503 uri = /omicron-zones
11752025-02-13T05:06:51.453ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "3df63878-29ae-49ca-89aa-73486f07bbcf", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:51 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "3df63878-29ae-49ca-89aa-73486f07bbcf" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 103.007799ms sled_address = [fd00:1122:3344:101::1]:12345
11762025-02-13T05:06:51.557ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
11772025-02-13T05:06:51.557ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11782025-02-13T05:06:51.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
11792025-02-13T05:06:51.569ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11802025-02-13T05:06:51.569ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11812025-02-13T05:06:51.580ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11822025-02-13T05:06:51.580ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11832025-02-13T05:06:51.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11842025-02-13T05:06:51.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11852025-02-13T05:06:51.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11862025-02-13T05:06:51.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11872025-02-13T05:06:51.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11882025-02-13T05:06:51.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11892025-02-13T05:06:51.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11902025-02-13T05:06:51.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
11912025-02-13T05:06:51.625ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
11922025-02-13T05:06:51.625ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
11932025-02-13T05:06:51.634ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11942025-02-13T05:06:51.634ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11952025-02-13T05:06:51.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
11962025-02-13T05:06:51.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11972025-02-13T05:06:51.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
11982025-02-13T05:06:51.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11992025-02-13T05:06:51.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12002025-02-13T05:06:51.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
12012025-02-13T05:06:51.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12022025-02-13T05:06:51.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
12032025-02-13T05:06:51.683ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12042025-02-13T05:06:51.683ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
12052025-02-13T05:06:51.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12062025-02-13T05:06:51.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
12072025-02-13T05:06:51.702ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12082025-02-13T05:06:51.702ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
12092025-02-13T05:06:51.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12102025-02-13T05:06:51.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
12112025-02-13T05:06:51.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12122025-02-13T05:06:51.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
12132025-02-13T05:06:51.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12142025-02-13T05:06:51.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
12152025-02-13T05:06:51.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12162025-02-13T05:06:51.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
12172025-02-13T05:06:51.750ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
12182025-02-13T05:06:51.750ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
12192025-02-13T05:06:51.750ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
12202025-02-13T05:06:51.751ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
12212025-02-13T05:06:51.751ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
12222025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
12232025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
12242025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
12252025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
12262025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
12272025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
12282025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
12292025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
12302025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
12312025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
12322025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
12332025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
12342025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
12352025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
12362025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
12372025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
12382025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
12392025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
12402025-02-13T05:06:51.845ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
12412025-02-13T05:06:51.845ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 288476 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 642ce620-9ff5-4c7d-bce1-56d976e9402a response_code = 503 uri = /omicron-zones
12422025-02-13T05:06:51.845ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "642ce620-9ff5-4c7d-bce1-56d976e9402a", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:51 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "642ce620-9ff5-4c7d-bce1-56d976e9402a" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 190.111911ms sled_address = [fd00:1122:3344:101::1]:12345
12432025-02-13T05:06:52.036ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
12442025-02-13T05:06:52.036ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
12452025-02-13T05:06:52.036ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
12462025-02-13T05:06:52.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
12472025-02-13T05:06:52.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12482025-02-13T05:06:52.056ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
12492025-02-13T05:06:52.056ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12502025-02-13T05:06:52.066ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
12512025-02-13T05:06:52.066ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12522025-02-13T05:06:52.076ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
12532025-02-13T05:06:52.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12542025-02-13T05:06:52.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
12552025-02-13T05:06:52.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12562025-02-13T05:06:52.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
12572025-02-13T05:06:52.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
12582025-02-13T05:06:52.105ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
12592025-02-13T05:06:52.105ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
12602025-02-13T05:06:52.114ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
12612025-02-13T05:06:52.114ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12622025-02-13T05:06:52.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
12632025-02-13T05:06:52.125ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12642025-02-13T05:06:52.135ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
12652025-02-13T05:06:52.135ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12662025-02-13T05:06:52.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12672025-02-13T05:06:52.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
12682025-02-13T05:06:52.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12692025-02-13T05:06:52.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
12702025-02-13T05:06:52.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12712025-02-13T05:06:52.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
12722025-02-13T05:06:52.175ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12732025-02-13T05:06:52.175ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
12742025-02-13T05:06:52.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12752025-02-13T05:06:52.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
12762025-02-13T05:06:52.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12772025-02-13T05:06:52.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
12782025-02-13T05:06:52.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12792025-02-13T05:06:52.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
12802025-02-13T05:06:52.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12812025-02-13T05:06:52.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
12822025-02-13T05:06:52.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
12832025-02-13T05:06:52.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
12842025-02-13T05:06:52.231ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
12852025-02-13T05:06:52.231ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
12862025-02-13T05:06:52.231ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
12872025-02-13T05:06:52.232ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
12882025-02-13T05:06:52.232ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
12892025-02-13T05:06:52.265ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 912.450887ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
12902025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
12912025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
12922025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
12932025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
12942025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
12952025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
12962025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
12972025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
12982025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
12992025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
13002025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
13012025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
13022025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
13032025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
13042025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
13052025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
13062025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
13072025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
13082025-02-13T05:06:52.327ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
13092025-02-13T05:06:52.327ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 291635 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 33fcb3f7-0e35-4b0f-8992-aee1505bb9a5 response_code = 503 uri = /omicron-zones
13102025-02-13T05:06:52.327ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "33fcb3f7-0e35-4b0f-8992-aee1505bb9a5", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:51 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "33fcb3f7-0e35-4b0f-8992-aee1505bb9a5" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 193.111723ms sled_address = [fd00:1122:3344:101::1]:12345
13112025-02-13T05:06:52.520ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
13122025-02-13T05:06:52.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13132025-02-13T05:06:52.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
13142025-02-13T05:06:52.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13152025-02-13T05:06:52.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13162025-02-13T05:06:52.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13172025-02-13T05:06:52.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13182025-02-13T05:06:52.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13192025-02-13T05:06:52.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
13202025-02-13T05:06:52.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13212025-02-13T05:06:52.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13222025-02-13T05:06:52.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13232025-02-13T05:06:52.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13242025-02-13T05:06:52.586ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13252025-02-13T05:06:52.586ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
13262025-02-13T05:06:52.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13272025-02-13T05:06:52.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
13282025-02-13T05:06:52.605ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13292025-02-13T05:06:52.605ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13302025-02-13T05:06:52.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13312025-02-13T05:06:52.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13322025-02-13T05:06:52.625ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
13332025-02-13T05:06:52.625ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13342025-02-13T05:06:52.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13352025-02-13T05:06:52.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
13362025-02-13T05:06:52.646ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13372025-02-13T05:06:52.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
13382025-02-13T05:06:52.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13392025-02-13T05:06:52.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
13402025-02-13T05:06:52.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13412025-02-13T05:06:52.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
13422025-02-13T05:06:52.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13432025-02-13T05:06:52.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
13442025-02-13T05:06:52.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13452025-02-13T05:06:52.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
13462025-02-13T05:06:52.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13472025-02-13T05:06:52.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
13482025-02-13T05:06:52.704ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13492025-02-13T05:06:52.704ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
13502025-02-13T05:06:52.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
13512025-02-13T05:06:52.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
13522025-02-13T05:06:52.723ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
13532025-02-13T05:06:52.723ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
13542025-02-13T05:06:52.723ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
13552025-02-13T05:06:52.723ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
13562025-02-13T05:06:52.723ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
13572025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
13582025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
13592025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
13602025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
13612025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
13622025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
13632025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
13642025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
13652025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
13662025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
13672025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
13682025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
13692025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
13702025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
13712025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
13722025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
13732025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
13742025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
13752025-02-13T05:06:52.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
13762025-02-13T05:06:52.818ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 298973 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 159d9eb6-abd9-4bbc-b45c-fdfdd49760ca response_code = 503 uri = /omicron-zones
13772025-02-13T05:06:52.818ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "159d9eb6-abd9-4bbc-b45c-fdfdd49760ca", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:51 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "159d9eb6-abd9-4bbc-b45c-fdfdd49760ca" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 168.977727ms sled_address = [fd00:1122:3344:101::1]:12345
13782025-02-13T05:06:52.988ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
13792025-02-13T05:06:52.989ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13802025-02-13T05:06:52.989ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
13812025-02-13T05:06:52.999ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13822025-02-13T05:06:53.000ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13832025-02-13T05:06:53.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13842025-02-13T05:06:53.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13852025-02-13T05:06:53.019ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13862025-02-13T05:06:53.019ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
13872025-02-13T05:06:53.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13882025-02-13T05:06:53.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13892025-02-13T05:06:53.040ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13902025-02-13T05:06:53.040ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13912025-02-13T05:06:53.049ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13922025-02-13T05:06:53.049ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
13932025-02-13T05:06:53.059ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
13942025-02-13T05:06:53.059ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
13952025-02-13T05:06:53.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13962025-02-13T05:06:53.070ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13972025-02-13T05:06:53.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
13982025-02-13T05:06:53.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13992025-02-13T05:06:53.088ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
14002025-02-13T05:06:53.088ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14012025-02-13T05:06:53.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14022025-02-13T05:06:53.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
14032025-02-13T05:06:53.107ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14042025-02-13T05:06:53.107ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
14052025-02-13T05:06:53.117ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14062025-02-13T05:06:53.117ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
14072025-02-13T05:06:53.126ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14082025-02-13T05:06:53.126ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
14092025-02-13T05:06:53.136ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14102025-02-13T05:06:53.136ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
14112025-02-13T05:06:53.144ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14122025-02-13T05:06:53.144ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
14132025-02-13T05:06:53.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14142025-02-13T05:06:53.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
14152025-02-13T05:06:53.163ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14162025-02-13T05:06:53.163ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
14172025-02-13T05:06:53.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14182025-02-13T05:06:53.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
14192025-02-13T05:06:53.181ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
14202025-02-13T05:06:53.181ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
14212025-02-13T05:06:53.181ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
14222025-02-13T05:06:53.182ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
14232025-02-13T05:06:53.182ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
14242025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
14252025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
14262025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
14272025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
14282025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
14292025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
14302025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
14312025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
14322025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
14332025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
14342025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
14352025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
14362025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
14372025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
14382025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
14392025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
14402025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
14412025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
14422025-02-13T05:06:53.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
14432025-02-13T05:06:53.280ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 291928 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 3318708a-a765-4052-84f3-fdec65b58bdb response_code = 503 uri = /omicron-zones
14442025-02-13T05:06:53.280ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "3318708a-a765-4052-84f3-fdec65b58bdb", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:53 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "3318708a-a765-4052-84f3-fdec65b58bdb" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 311.619008ms sled_address = [fd00:1122:3344:101::1]:12345
14452025-02-13T05:06:53.591ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
14462025-02-13T05:06:53.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
14472025-02-13T05:06:53.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
14482025-02-13T05:06:53.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
14492025-02-13T05:06:53.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14502025-02-13T05:06:53.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
14512025-02-13T05:06:53.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14522025-02-13T05:06:53.623ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
14532025-02-13T05:06:53.623ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
14542025-02-13T05:06:53.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
14552025-02-13T05:06:53.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14562025-02-13T05:06:53.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
14572025-02-13T05:06:53.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14582025-02-13T05:06:53.651ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
14592025-02-13T05:06:53.651ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
14602025-02-13T05:06:53.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
14612025-02-13T05:06:53.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
14622025-02-13T05:06:53.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
14632025-02-13T05:06:53.668ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14642025-02-13T05:06:53.677ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
14652025-02-13T05:06:53.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14662025-02-13T05:06:53.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
14672025-02-13T05:06:53.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14682025-02-13T05:06:53.697ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14692025-02-13T05:06:53.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
14702025-02-13T05:06:53.706ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14712025-02-13T05:06:53.706ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
14722025-02-13T05:06:53.715ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14732025-02-13T05:06:53.715ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
14742025-02-13T05:06:53.724ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14752025-02-13T05:06:53.725ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
14762025-02-13T05:06:53.735ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14772025-02-13T05:06:53.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
14782025-02-13T05:06:53.744ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14792025-02-13T05:06:53.744ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
14802025-02-13T05:06:53.753ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14812025-02-13T05:06:53.753ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
14822025-02-13T05:06:53.761ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14832025-02-13T05:06:53.761ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
14842025-02-13T05:06:53.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
14852025-02-13T05:06:53.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
14862025-02-13T05:06:53.778ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
14872025-02-13T05:06:53.778ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
14882025-02-13T05:06:53.778ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
14892025-02-13T05:06:53.778ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
14902025-02-13T05:06:53.778ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
14912025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
14922025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
14932025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
14942025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
14952025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
14962025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
14972025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
14982025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
14992025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
15002025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
15012025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
15022025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
15032025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
15042025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
15052025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
15062025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
15072025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
15082025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
15092025-02-13T05:06:53.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
15102025-02-13T05:06:53.872ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 281482 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = e0d13cb6-c0ba-4d3c-916d-b9d30ac0b2c1 response_code = 503 uri = /omicron-zones
15112025-02-13T05:06:53.872ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e0d13cb6-c0ba-4d3c-916d-b9d30ac0b2c1", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:52 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "e0d13cb6-c0ba-4d3c-916d-b9d30ac0b2c1" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 305.24574ms sled_address = [fd00:1122:3344:101::1]:12345
15122025-02-13T05:06:54.178ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
15132025-02-13T05:06:54.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15142025-02-13T05:06:54.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
15152025-02-13T05:06:54.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15162025-02-13T05:06:54.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15172025-02-13T05:06:54.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15182025-02-13T05:06:54.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15192025-02-13T05:06:54.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15202025-02-13T05:06:54.207ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
15212025-02-13T05:06:54.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15222025-02-13T05:06:54.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15232025-02-13T05:06:54.225ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15242025-02-13T05:06:54.225ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15252025-02-13T05:06:54.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15262025-02-13T05:06:54.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
15272025-02-13T05:06:54.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15282025-02-13T05:06:54.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
15292025-02-13T05:06:54.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15302025-02-13T05:06:54.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15312025-02-13T05:06:54.259ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15322025-02-13T05:06:54.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15332025-02-13T05:06:54.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
15342025-02-13T05:06:54.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15352025-02-13T05:06:54.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15362025-02-13T05:06:54.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
15372025-02-13T05:06:54.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15382025-02-13T05:06:54.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
15392025-02-13T05:06:54.294ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15402025-02-13T05:06:54.294ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
15412025-02-13T05:06:54.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15422025-02-13T05:06:54.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
15432025-02-13T05:06:54.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15442025-02-13T05:06:54.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
15452025-02-13T05:06:54.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15462025-02-13T05:06:54.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
15472025-02-13T05:06:54.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15482025-02-13T05:06:54.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
15492025-02-13T05:06:54.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15502025-02-13T05:06:54.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
15512025-02-13T05:06:54.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
15522025-02-13T05:06:54.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
15532025-02-13T05:06:54.355ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
15542025-02-13T05:06:54.355ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
15552025-02-13T05:06:54.355ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
15562025-02-13T05:06:54.356ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
15572025-02-13T05:06:54.356ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
15582025-02-13T05:06:54.449ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
15592025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
15602025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
15612025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
15622025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
15632025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
15642025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
15652025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
15662025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
15672025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
15682025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
15692025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
15702025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
15712025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
15722025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
15732025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
15742025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
15752025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
15762025-02-13T05:06:54.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
15772025-02-13T05:06:54.450ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 272473 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = a70d229f-9f2e-4fbe-b4b7-6afc49c312c3 response_code = 503 uri = /omicron-zones
15782025-02-13T05:06:54.450ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "a70d229f-9f2e-4fbe-b4b7-6afc49c312c3", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:54 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "a70d229f-9f2e-4fbe-b4b7-6afc49c312c3" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 462.676052ms sled_address = [fd00:1122:3344:101::1]:12345
15792025-02-13T05:06:54.612ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36129
15802025-02-13T05:06:54.612ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 30 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36129 req_id = 394691e3-f1e7-4f23-aa2d-31299bbfc8b4 response_code = 200 uri = /baseboard
15812025-02-13T05:06:54.913ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
15822025-02-13T05:06:54.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15832025-02-13T05:06:54.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
15842025-02-13T05:06:54.924ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15852025-02-13T05:06:54.924ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15862025-02-13T05:06:54.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15872025-02-13T05:06:54.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15882025-02-13T05:06:54.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15892025-02-13T05:06:54.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
15902025-02-13T05:06:54.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15912025-02-13T05:06:54.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15922025-02-13T05:06:54.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15932025-02-13T05:06:54.964ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15942025-02-13T05:06:54.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15952025-02-13T05:06:54.975ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
15962025-02-13T05:06:54.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
15972025-02-13T05:06:54.984ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
15982025-02-13T05:06:54.994ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
15992025-02-13T05:06:54.994ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16002025-02-13T05:06:55.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
16012025-02-13T05:06:55.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16022025-02-13T05:06:55.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
16032025-02-13T05:06:55.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16042025-02-13T05:06:55.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16052025-02-13T05:06:55.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
16062025-02-13T05:06:55.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16072025-02-13T05:06:55.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
16082025-02-13T05:06:55.041ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16092025-02-13T05:06:55.041ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
16102025-02-13T05:06:55.051ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16112025-02-13T05:06:55.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
16122025-02-13T05:06:55.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16132025-02-13T05:06:55.060ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
16142025-02-13T05:06:55.070ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16152025-02-13T05:06:55.070ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
16162025-02-13T05:06:55.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16172025-02-13T05:06:55.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
16182025-02-13T05:06:55.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16192025-02-13T05:06:55.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
16202025-02-13T05:06:55.098ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16212025-02-13T05:06:55.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
16222025-02-13T05:06:55.106ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
16232025-02-13T05:06:55.107ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
16242025-02-13T05:06:55.107ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
16252025-02-13T05:06:55.107ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
16262025-02-13T05:06:55.107ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
16272025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
16282025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
16292025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
16302025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
16312025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
16322025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
16332025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
16342025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
16352025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
16362025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
16372025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
16382025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
16392025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
16402025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
16412025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
16422025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
16432025-02-13T05:06:55.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
16442025-02-13T05:06:55.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
16452025-02-13T05:06:55.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
16462025-02-13T05:06:55.200ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 287373 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = ee84af8b-5301-4195-9fd3-f1d1d0689ebf response_code = 503 uri = /omicron-zones
16472025-02-13T05:06:55.200ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "ee84af8b-5301-4195-9fd3-f1d1d0689ebf", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:54 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "ee84af8b-5301-4195-9fd3-f1d1d0689ebf" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 409.065767ms sled_address = [fd00:1122:3344:101::1]:12345
16482025-02-13T05:06:55.609ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
16492025-02-13T05:06:55.609ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
16502025-02-13T05:06:55.609ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
16512025-02-13T05:06:55.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
16522025-02-13T05:06:55.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16532025-02-13T05:06:55.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
16542025-02-13T05:06:55.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16552025-02-13T05:06:55.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
16562025-02-13T05:06:55.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
16572025-02-13T05:06:55.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
16582025-02-13T05:06:55.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
16592025-02-13T05:06:55.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
16602025-02-13T05:06:55.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
16612025-02-13T05:06:55.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
16622025-02-13T05:06:55.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
16632025-02-13T05:06:55.683ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
16642025-02-13T05:06:55.683ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
16652025-02-13T05:06:55.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
16662025-02-13T05:06:55.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16672025-02-13T05:06:55.705ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
16682025-02-13T05:06:55.705ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16692025-02-13T05:06:55.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
16702025-02-13T05:06:55.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16712025-02-13T05:06:55.728ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16722025-02-13T05:06:55.728ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
16732025-02-13T05:06:55.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16742025-02-13T05:06:55.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
16752025-02-13T05:06:55.753ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16762025-02-13T05:06:55.753ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
16772025-02-13T05:06:55.764ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16782025-02-13T05:06:55.764ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
16792025-02-13T05:06:55.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16802025-02-13T05:06:55.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
16812025-02-13T05:06:55.785ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16822025-02-13T05:06:55.785ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
16832025-02-13T05:06:55.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16842025-02-13T05:06:55.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
16852025-02-13T05:06:55.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16862025-02-13T05:06:55.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
16872025-02-13T05:06:55.815ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
16882025-02-13T05:06:55.815ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
16892025-02-13T05:06:55.826ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
16902025-02-13T05:06:55.826ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
16912025-02-13T05:06:55.826ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
16922025-02-13T05:06:55.826ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
16932025-02-13T05:06:55.826ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
16942025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
16952025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
16962025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
16972025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
16982025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
16992025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
17002025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
17012025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
17022025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
17032025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
17042025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
17052025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
17062025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
17072025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
17082025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
17092025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
17102025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
17112025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
17122025-02-13T05:06:55.927ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
17132025-02-13T05:06:55.927ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 319373 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 4a4198d5-1a9d-488a-877e-9904e2200a42 response_code = 503 uri = /omicron-zones
17142025-02-13T05:06:55.928ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "4a4198d5-1a9d-488a-877e-9904e2200a42", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:55 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "4a4198d5-1a9d-488a-877e-9904e2200a42" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 320.555486ms sled_address = [fd00:1122:3344:101::1]:12345
17152025-02-13T05:06:56.248ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
17162025-02-13T05:06:56.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17172025-02-13T05:06:56.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
17182025-02-13T05:06:56.260ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17192025-02-13T05:06:56.260ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17202025-02-13T05:06:56.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17212025-02-13T05:06:56.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17222025-02-13T05:06:56.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17232025-02-13T05:06:56.283ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
17242025-02-13T05:06:56.294ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17252025-02-13T05:06:56.294ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17262025-02-13T05:06:56.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17272025-02-13T05:06:56.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17282025-02-13T05:06:56.318ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17292025-02-13T05:06:56.318ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
17302025-02-13T05:06:56.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17312025-02-13T05:06:56.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
17322025-02-13T05:06:56.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17332025-02-13T05:06:56.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17342025-02-13T05:06:56.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17352025-02-13T05:06:56.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17362025-02-13T05:06:56.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
17372025-02-13T05:06:56.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17382025-02-13T05:06:56.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17392025-02-13T05:06:56.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
17402025-02-13T05:06:56.377ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17412025-02-13T05:06:56.377ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
17422025-02-13T05:06:56.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17432025-02-13T05:06:56.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17442025-02-13T05:06:56.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17452025-02-13T05:06:56.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
17462025-02-13T05:06:56.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17472025-02-13T05:06:56.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
17482025-02-13T05:06:56.418ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17492025-02-13T05:06:56.418ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
17502025-02-13T05:06:56.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17512025-02-13T05:06:56.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
17522025-02-13T05:06:56.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17532025-02-13T05:06:56.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
17542025-02-13T05:06:56.450ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
17552025-02-13T05:06:56.450ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
17562025-02-13T05:06:56.462ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
17572025-02-13T05:06:56.462ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
17582025-02-13T05:06:56.462ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
17592025-02-13T05:06:56.462ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
17602025-02-13T05:06:56.462ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
17612025-02-13T05:06:56.562ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
17622025-02-13T05:06:56.562ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
17632025-02-13T05:06:56.562ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
17642025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
17652025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
17662025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
17672025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
17682025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
17692025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
17702025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
17712025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
17722025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
17732025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
17742025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
17752025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
17762025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
17772025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
17782025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
17792025-02-13T05:06:56.563ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
17802025-02-13T05:06:56.563ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 315101 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 34c945a5-ecc4-4dea-9595-337a869b70a2 response_code = 503 uri = /omicron-zones
17812025-02-13T05:06:56.563ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "34c945a5-ecc4-4dea-9595-337a869b70a2", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:55 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "34c945a5-ecc4-4dea-9595-337a869b70a2" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 374.407846ms sled_address = [fd00:1122:3344:101::1]:12345
17822025-02-13T05:06:56.937ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
17832025-02-13T05:06:56.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17842025-02-13T05:06:56.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
17852025-02-13T05:06:56.950ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17862025-02-13T05:06:56.950ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17872025-02-13T05:06:56.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
17882025-02-13T05:06:56.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17892025-02-13T05:06:56.971ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17902025-02-13T05:06:56.971ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
17912025-02-13T05:06:56.981ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17922025-02-13T05:06:56.981ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17932025-02-13T05:06:56.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17942025-02-13T05:06:56.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17952025-02-13T05:06:57.001ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17962025-02-13T05:06:57.001ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
17972025-02-13T05:06:57.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
17982025-02-13T05:06:57.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
17992025-02-13T05:06:57.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
18002025-02-13T05:06:57.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18012025-02-13T05:06:57.032ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
18022025-02-13T05:06:57.032ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18032025-02-13T05:06:57.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
18042025-02-13T05:06:57.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18052025-02-13T05:06:57.052ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18062025-02-13T05:06:57.052ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
18072025-02-13T05:06:57.065ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18082025-02-13T05:06:57.065ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
18092025-02-13T05:06:57.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18102025-02-13T05:06:57.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
18112025-02-13T05:06:57.084ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18122025-02-13T05:06:57.084ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
18132025-02-13T05:06:57.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18142025-02-13T05:06:57.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
18152025-02-13T05:06:57.107ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18162025-02-13T05:06:57.107ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
18172025-02-13T05:06:57.119ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18182025-02-13T05:06:57.119ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
18192025-02-13T05:06:57.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18202025-02-13T05:06:57.131ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
18212025-02-13T05:06:57.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18222025-02-13T05:06:57.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
18232025-02-13T05:06:57.151ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
18242025-02-13T05:06:57.151ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
18252025-02-13T05:06:57.151ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
18262025-02-13T05:06:57.152ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
18272025-02-13T05:06:57.152ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
18282025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
18292025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
18302025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
18312025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
18322025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
18332025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
18342025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
18352025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
18362025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
18372025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
18382025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
18392025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
18402025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
18412025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
18422025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
18432025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
18442025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
18452025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
18462025-02-13T05:06:57.246ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
18472025-02-13T05:06:57.246ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 309807 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 3062ac25-9881-49b9-b30a-2a929d9bcd76 response_code = 503 uri = /omicron-zones
18482025-02-13T05:06:57.246ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "3062ac25-9881-49b9-b30a-2a929d9bcd76", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:57 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "3062ac25-9881-49b9-b30a-2a929d9bcd76" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 432.639959ms sled_address = [fd00:1122:3344:101::1]:12345
18492025-02-13T05:06:57.679ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
18502025-02-13T05:06:57.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
18512025-02-13T05:06:57.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
18522025-02-13T05:06:57.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
18532025-02-13T05:06:57.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18542025-02-13T05:06:57.703ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
18552025-02-13T05:06:57.703ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18562025-02-13T05:06:57.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
18572025-02-13T05:06:57.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
18582025-02-13T05:06:57.725ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
18592025-02-13T05:06:57.725ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
18602025-02-13T05:06:57.735ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
18612025-02-13T05:06:57.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
18622025-02-13T05:06:57.746ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
18632025-02-13T05:06:57.746ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
18642025-02-13T05:06:57.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
18652025-02-13T05:06:57.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
18662025-02-13T05:06:57.766ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
18672025-02-13T05:06:57.766ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18682025-02-13T05:06:57.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
18692025-02-13T05:06:57.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18702025-02-13T05:06:57.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
18712025-02-13T05:06:57.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18722025-02-13T05:06:57.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18732025-02-13T05:06:57.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
18742025-02-13T05:06:57.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18752025-02-13T05:06:57.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
18762025-02-13T05:06:57.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18772025-02-13T05:06:57.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
18782025-02-13T05:06:57.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18792025-02-13T05:06:57.835ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
18802025-02-13T05:06:57.844ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18812025-02-13T05:06:57.844ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
18822025-02-13T05:06:57.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18832025-02-13T05:06:57.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
18842025-02-13T05:06:57.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18852025-02-13T05:06:57.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
18862025-02-13T05:06:57.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18872025-02-13T05:06:57.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
18882025-02-13T05:06:57.887ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
18892025-02-13T05:06:57.887ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
18902025-02-13T05:06:57.897ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
18912025-02-13T05:06:57.897ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
18922025-02-13T05:06:57.897ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
18932025-02-13T05:06:57.897ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
18942025-02-13T05:06:57.897ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
18952025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
18962025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
18972025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
18982025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
18992025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
19002025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
19012025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
19022025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
19032025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
19042025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
19052025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
19062025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
19072025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
19082025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
19092025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
19102025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
19112025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
19122025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
19132025-02-13T05:06:57.996ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
19142025-02-13T05:06:57.996ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 317027 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 39a0209f-5e74-4ed2-9ee3-9208e7f35457 response_code = 503 uri = /omicron-zones
19152025-02-13T05:06:57.996ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "39a0209f-5e74-4ed2-9ee3-9208e7f35457", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:57 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "39a0209f-5e74-4ed2-9ee3-9208e7f35457" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 1.074358496s sled_address = [fd00:1122:3344:101::1]:12345
19162025-02-13T05:06:59.069ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
19172025-02-13T05:06:59.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19182025-02-13T05:06:59.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
19192025-02-13T05:06:59.081ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19202025-02-13T05:06:59.081ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19212025-02-13T05:06:59.090ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19222025-02-13T05:06:59.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19232025-02-13T05:06:59.100ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19242025-02-13T05:06:59.100ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
19252025-02-13T05:06:59.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19262025-02-13T05:06:59.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19272025-02-13T05:06:59.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19282025-02-13T05:06:59.120ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19292025-02-13T05:06:59.129ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19302025-02-13T05:06:59.129ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
19312025-02-13T05:06:59.140ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19322025-02-13T05:06:59.140ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
19332025-02-13T05:06:59.149ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19342025-02-13T05:06:59.149ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19352025-02-13T05:06:59.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19362025-02-13T05:06:59.158ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19372025-02-13T05:06:59.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
19382025-02-13T05:06:59.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19392025-02-13T05:06:59.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19402025-02-13T05:06:59.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
19412025-02-13T05:06:59.187ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19422025-02-13T05:06:59.187ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
19432025-02-13T05:06:59.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19442025-02-13T05:06:59.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
19452025-02-13T05:06:59.205ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19462025-02-13T05:06:59.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
19472025-02-13T05:06:59.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19482025-02-13T05:06:59.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
19492025-02-13T05:06:59.224ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19502025-02-13T05:06:59.224ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
19512025-02-13T05:06:59.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19522025-02-13T05:06:59.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
19532025-02-13T05:06:59.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19542025-02-13T05:06:59.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
19552025-02-13T05:06:59.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
19562025-02-13T05:06:59.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
19572025-02-13T05:06:59.260ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
19582025-02-13T05:06:59.260ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
19592025-02-13T05:06:59.260ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
19602025-02-13T05:06:59.260ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
19612025-02-13T05:06:59.260ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
19622025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
19632025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
19642025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
19652025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
19662025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
19672025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
19682025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
19692025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
19702025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
19712025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
19722025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
19732025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
19742025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
19752025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
19762025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
19772025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
19782025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
19792025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
19802025-02-13T05:06:59.356ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
19812025-02-13T05:06:59.356ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 287841 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 7f671b5c-0e60-42e4-96f0-999341abdc6f response_code = 503 uri = /omicron-zones
19822025-02-13T05:06:59.356ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "7f671b5c-0e60-42e4-96f0-999341abdc6f", "content-length": "133", "date": "Thu, 13 Feb 2025 05:06:59 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "7f671b5c-0e60-42e4-96f0-999341abdc6f" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 1.172657338s sled_address = [fd00:1122:3344:101::1]:12345
19832025-02-13T05:07:00.527ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
19842025-02-13T05:07:00.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19852025-02-13T05:07:00.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
19862025-02-13T05:07:00.539ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19872025-02-13T05:07:00.539ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19882025-02-13T05:07:00.549ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
19892025-02-13T05:07:00.549ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19902025-02-13T05:07:00.561ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19912025-02-13T05:07:00.561ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
19922025-02-13T05:07:00.572ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19932025-02-13T05:07:00.572ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19942025-02-13T05:07:00.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19952025-02-13T05:07:00.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
19962025-02-13T05:07:00.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19972025-02-13T05:07:00.593ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
19982025-02-13T05:07:00.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
19992025-02-13T05:07:00.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
20002025-02-13T05:07:00.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
20012025-02-13T05:07:00.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20022025-02-13T05:07:00.625ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
20032025-02-13T05:07:00.625ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20042025-02-13T05:07:00.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
20052025-02-13T05:07:00.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20062025-02-13T05:07:00.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20072025-02-13T05:07:00.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
20082025-02-13T05:07:00.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20092025-02-13T05:07:00.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
20102025-02-13T05:07:00.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20112025-02-13T05:07:00.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
20122025-02-13T05:07:00.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20132025-02-13T05:07:00.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
20142025-02-13T05:07:00.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20152025-02-13T05:07:00.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
20162025-02-13T05:07:00.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20172025-02-13T05:07:00.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
20182025-02-13T05:07:00.705ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20192025-02-13T05:07:00.705ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
20202025-02-13T05:07:00.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20212025-02-13T05:07:00.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
20222025-02-13T05:07:00.725ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20232025-02-13T05:07:00.725ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
20242025-02-13T05:07:00.735ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
20252025-02-13T05:07:00.735ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
20262025-02-13T05:07:00.735ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
20272025-02-13T05:07:00.735ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
20282025-02-13T05:07:00.735ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
20292025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
20302025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
20312025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
20322025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
20332025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
20342025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
20352025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
20362025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
20372025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
20382025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
20392025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
20402025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
20412025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
20422025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
20432025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
20442025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
20452025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
20462025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
20472025-02-13T05:07:00.834ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
20482025-02-13T05:07:00.834ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 307925 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = d368a66a-22b4-43aa-8a95-0f595448b47d response_code = 503 uri = /omicron-zones
20492025-02-13T05:07:00.834ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "d368a66a-22b4-43aa-8a95-0f595448b47d", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:00 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "d368a66a-22b4-43aa-8a95-0f595448b47d" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 1.593308731s sled_address = [fd00:1122:3344:101::1]:12345
20502025-02-13T05:07:02.424ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
20512025-02-13T05:07:02.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
20522025-02-13T05:07:02.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
20532025-02-13T05:07:02.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
20542025-02-13T05:07:02.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20552025-02-13T05:07:02.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
20562025-02-13T05:07:02.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20572025-02-13T05:07:02.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
20582025-02-13T05:07:02.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
20592025-02-13T05:07:02.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
20602025-02-13T05:07:02.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
20612025-02-13T05:07:02.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
20622025-02-13T05:07:02.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
20632025-02-13T05:07:02.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
20642025-02-13T05:07:02.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
20652025-02-13T05:07:02.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
20662025-02-13T05:07:02.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
20672025-02-13T05:07:02.512ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
20682025-02-13T05:07:02.512ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20692025-02-13T05:07:02.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
20702025-02-13T05:07:02.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20712025-02-13T05:07:02.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
20722025-02-13T05:07:02.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20732025-02-13T05:07:02.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20742025-02-13T05:07:02.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
20752025-02-13T05:07:02.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20762025-02-13T05:07:02.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
20772025-02-13T05:07:02.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20782025-02-13T05:07:02.563ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
20792025-02-13T05:07:02.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20802025-02-13T05:07:02.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
20812025-02-13T05:07:02.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20822025-02-13T05:07:02.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
20832025-02-13T05:07:02.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20842025-02-13T05:07:02.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
20852025-02-13T05:07:02.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20862025-02-13T05:07:02.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
20872025-02-13T05:07:02.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20882025-02-13T05:07:02.618ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
20892025-02-13T05:07:02.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
20902025-02-13T05:07:02.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
20912025-02-13T05:07:02.638ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
20922025-02-13T05:07:02.638ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
20932025-02-13T05:07:02.638ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
20942025-02-13T05:07:02.639ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
20952025-02-13T05:07:02.639ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
20962025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
20972025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
20982025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
20992025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
21002025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
21012025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
21022025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
21032025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
21042025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
21052025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
21062025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
21072025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
21082025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
21092025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
21102025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
21112025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
21122025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
21132025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
21142025-02-13T05:07:02.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
21152025-02-13T05:07:02.740ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 317029 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 00d6b214-b817-4584-9ca6-144c0cbc0ed9 response_code = 503 uri = /omicron-zones
21162025-02-13T05:07:02.740ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "00d6b214-b817-4584-9ca6-144c0cbc0ed9", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:02 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "00d6b214-b817-4584-9ca6-144c0cbc0ed9" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 2.016209281s sled_address = [fd00:1122:3344:101::1]:12345
21172025-02-13T05:07:04.752ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
21182025-02-13T05:07:04.752ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21192025-02-13T05:07:04.752ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
21202025-02-13T05:07:04.764ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21212025-02-13T05:07:04.764ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21222025-02-13T05:07:04.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21232025-02-13T05:07:04.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21242025-02-13T05:07:04.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21252025-02-13T05:07:04.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
21262025-02-13T05:07:04.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21272025-02-13T05:07:04.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21282025-02-13T05:07:04.805ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21292025-02-13T05:07:04.805ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21302025-02-13T05:07:04.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21312025-02-13T05:07:04.816ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
21322025-02-13T05:07:04.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21332025-02-13T05:07:04.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
21342025-02-13T05:07:04.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21352025-02-13T05:07:04.835ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21362025-02-13T05:07:04.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21372025-02-13T05:07:04.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21382025-02-13T05:07:04.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
21392025-02-13T05:07:04.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21402025-02-13T05:07:04.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21412025-02-13T05:07:04.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
21422025-02-13T05:07:04.874ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21432025-02-13T05:07:04.874ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
21442025-02-13T05:07:04.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21452025-02-13T05:07:04.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
21462025-02-13T05:07:04.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21472025-02-13T05:07:04.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
21482025-02-13T05:07:04.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21492025-02-13T05:07:04.903ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
21502025-02-13T05:07:04.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21512025-02-13T05:07:04.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
21522025-02-13T05:07:04.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21532025-02-13T05:07:04.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
21542025-02-13T05:07:04.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21552025-02-13T05:07:04.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
21562025-02-13T05:07:04.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
21572025-02-13T05:07:04.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
21582025-02-13T05:07:04.945ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
21592025-02-13T05:07:04.945ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
21602025-02-13T05:07:04.945ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
21612025-02-13T05:07:04.945ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
21622025-02-13T05:07:04.945ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
21632025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
21642025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
21652025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
21662025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
21672025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
21682025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
21692025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
21702025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
21712025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
21722025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
21732025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
21742025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
21752025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
21762025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
21772025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
21782025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
21792025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
21802025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
21812025-02-13T05:07:05.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
21822025-02-13T05:07:05.041ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 290110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = ecfab51e-2243-4886-ad3b-f14334586112 response_code = 503 uri = /omicron-zones
21832025-02-13T05:07:05.041ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "ecfab51e-2243-4886-ad3b-f14334586112", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:05 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "ecfab51e-2243-4886-ad3b-f14334586112" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 1.087500643s sled_address = [fd00:1122:3344:101::1]:12345
21842025-02-13T05:07:06.127ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
21852025-02-13T05:07:06.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21862025-02-13T05:07:06.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
21872025-02-13T05:07:06.140ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21882025-02-13T05:07:06.140ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21892025-02-13T05:07:06.152ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
21902025-02-13T05:07:06.152ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21912025-02-13T05:07:06.163ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21922025-02-13T05:07:06.163ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
21932025-02-13T05:07:06.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21942025-02-13T05:07:06.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21952025-02-13T05:07:06.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21962025-02-13T05:07:06.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
21972025-02-13T05:07:06.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
21982025-02-13T05:07:06.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
21992025-02-13T05:07:06.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
22002025-02-13T05:07:06.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
22012025-02-13T05:07:06.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
22022025-02-13T05:07:06.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22032025-02-13T05:07:06.223ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
22042025-02-13T05:07:06.223ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22052025-02-13T05:07:06.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
22062025-02-13T05:07:06.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22072025-02-13T05:07:06.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22082025-02-13T05:07:06.244ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
22092025-02-13T05:07:06.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22102025-02-13T05:07:06.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
22112025-02-13T05:07:06.263ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22122025-02-13T05:07:06.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
22132025-02-13T05:07:06.273ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22142025-02-13T05:07:06.273ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
22152025-02-13T05:07:06.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22162025-02-13T05:07:06.283ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
22172025-02-13T05:07:06.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22182025-02-13T05:07:06.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
22192025-02-13T05:07:06.303ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22202025-02-13T05:07:06.303ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
22212025-02-13T05:07:06.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22222025-02-13T05:07:06.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
22232025-02-13T05:07:06.324ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22242025-02-13T05:07:06.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
22252025-02-13T05:07:06.333ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
22262025-02-13T05:07:06.334ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
22272025-02-13T05:07:06.334ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
22282025-02-13T05:07:06.334ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
22292025-02-13T05:07:06.334ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
22302025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
22312025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
22322025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
22332025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
22342025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
22352025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
22362025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
22372025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
22382025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
22392025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
22402025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
22412025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
22422025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
22432025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
22442025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
22452025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
22462025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
22472025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
22482025-02-13T05:07:06.432ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
22492025-02-13T05:07:06.432ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 305344 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 560b275a-a6c6-4a73-8e7e-b11afe07190a response_code = 503 uri = /omicron-zones
22502025-02-13T05:07:06.432ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "560b275a-a6c6-4a73-8e7e-b11afe07190a", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:06 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "560b275a-a6c6-4a73-8e7e-b11afe07190a" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 3.327275851s sled_address = [fd00:1122:3344:101::1]:12345
22512025-02-13T05:07:08.213ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.684224964s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
22522025-02-13T05:07:09.752ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
22532025-02-13T05:07:09.752ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
22542025-02-13T05:07:09.752ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
22552025-02-13T05:07:09.764ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
22562025-02-13T05:07:09.764ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22572025-02-13T05:07:09.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
22582025-02-13T05:07:09.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22592025-02-13T05:07:09.785ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
22602025-02-13T05:07:09.785ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
22612025-02-13T05:07:09.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
22622025-02-13T05:07:09.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
22632025-02-13T05:07:09.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
22642025-02-13T05:07:09.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
22652025-02-13T05:07:09.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
22662025-02-13T05:07:09.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
22672025-02-13T05:07:09.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
22682025-02-13T05:07:09.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
22692025-02-13T05:07:09.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
22702025-02-13T05:07:09.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22712025-02-13T05:07:09.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
22722025-02-13T05:07:09.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22732025-02-13T05:07:09.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
22742025-02-13T05:07:09.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22752025-02-13T05:07:09.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22762025-02-13T05:07:09.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
22772025-02-13T05:07:09.874ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22782025-02-13T05:07:09.874ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
22792025-02-13T05:07:09.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22802025-02-13T05:07:09.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
22812025-02-13T05:07:09.896ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22822025-02-13T05:07:09.896ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
22832025-02-13T05:07:09.906ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22842025-02-13T05:07:09.906ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
22852025-02-13T05:07:09.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22862025-02-13T05:07:09.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
22872025-02-13T05:07:09.925ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22882025-02-13T05:07:09.925ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
22892025-02-13T05:07:09.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22902025-02-13T05:07:09.934ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
22912025-02-13T05:07:09.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
22922025-02-13T05:07:09.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
22932025-02-13T05:07:09.953ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
22942025-02-13T05:07:09.953ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
22952025-02-13T05:07:09.953ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
22962025-02-13T05:07:09.953ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
22972025-02-13T05:07:09.953ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
22982025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
22992025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
23002025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
23012025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
23022025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
23032025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
23042025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
23052025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
23062025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
23072025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
23082025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
23092025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
23102025-02-13T05:07:10.048ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
23112025-02-13T05:07:10.049ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
23122025-02-13T05:07:10.049ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
23132025-02-13T05:07:10.049ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
23142025-02-13T05:07:10.049ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
23152025-02-13T05:07:10.049ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
23162025-02-13T05:07:10.049ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
23172025-02-13T05:07:10.049ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 297710 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 123797c2-7653-4eba-84f5-dcc562e5281a response_code = 503 uri = /omicron-zones
23182025-02-13T05:07:10.049ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "123797c2-7653-4eba-84f5-dcc562e5281a", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:09 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "123797c2-7653-4eba-84f5-dcc562e5281a" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 2.435558554s sled_address = [fd00:1122:3344:101::1]:12345
23192025-02-13T05:07:12.479ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
23202025-02-13T05:07:12.479ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23212025-02-13T05:07:12.479ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
23222025-02-13T05:07:12.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23232025-02-13T05:07:12.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23242025-02-13T05:07:12.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23252025-02-13T05:07:12.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23262025-02-13T05:07:12.515ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23272025-02-13T05:07:12.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
23282025-02-13T05:07:12.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23292025-02-13T05:07:12.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
23302025-02-13T05:07:12.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23312025-02-13T05:07:12.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
23322025-02-13T05:07:12.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23332025-02-13T05:07:12.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
23342025-02-13T05:07:12.557ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23352025-02-13T05:07:12.557ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
23362025-02-13T05:07:12.567ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23372025-02-13T05:07:12.567ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23382025-02-13T05:07:12.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23392025-02-13T05:07:12.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23402025-02-13T05:07:12.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
23412025-02-13T05:07:12.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
23422025-02-13T05:07:12.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23432025-02-13T05:07:12.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
23442025-02-13T05:07:12.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23452025-02-13T05:07:12.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
23462025-02-13T05:07:12.617ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23472025-02-13T05:07:12.617ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
23482025-02-13T05:07:12.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23492025-02-13T05:07:12.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
23502025-02-13T05:07:12.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23512025-02-13T05:07:12.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
23522025-02-13T05:07:12.646ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23532025-02-13T05:07:12.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
23542025-02-13T05:07:12.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23552025-02-13T05:07:12.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
23562025-02-13T05:07:12.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23572025-02-13T05:07:12.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
23582025-02-13T05:07:12.675ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
23592025-02-13T05:07:12.675ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
23602025-02-13T05:07:12.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
23612025-02-13T05:07:12.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
23622025-02-13T05:07:12.685ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
23632025-02-13T05:07:12.685ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
23642025-02-13T05:07:12.685ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
23652025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
23662025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
23672025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
23682025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
23692025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
23702025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
23712025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
23722025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
23732025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
23742025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
23752025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
23762025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
23772025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
23782025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
23792025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
23802025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
23812025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
23822025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
23832025-02-13T05:07:12.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
23842025-02-13T05:07:12.785ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 307330 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 0bbbdbc1-9b87-4dff-9744-b6db471fe702 response_code = 503 uri = /omicron-zones
23852025-02-13T05:07:12.785ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "0bbbdbc1-9b87-4dff-9744-b6db471fe702", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:12 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "0bbbdbc1-9b87-4dff-9744-b6db471fe702" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 2.451483075s sled_address = [fd00:1122:3344:101::1]:12345
23862025-02-13T05:07:15.232ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
23872025-02-13T05:07:15.232ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23882025-02-13T05:07:15.232ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
23892025-02-13T05:07:15.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23902025-02-13T05:07:15.244ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23912025-02-13T05:07:15.255ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
23922025-02-13T05:07:15.255ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23932025-02-13T05:07:15.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23942025-02-13T05:07:15.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
23952025-02-13T05:07:15.276ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23962025-02-13T05:07:15.276ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
23972025-02-13T05:07:15.286ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
23982025-02-13T05:07:15.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
23992025-02-13T05:07:15.295ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
24002025-02-13T05:07:15.295ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
24012025-02-13T05:07:15.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
24022025-02-13T05:07:15.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
24032025-02-13T05:07:15.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
24042025-02-13T05:07:15.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24052025-02-13T05:07:15.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
24062025-02-13T05:07:15.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24072025-02-13T05:07:15.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
24082025-02-13T05:07:15.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24092025-02-13T05:07:15.345ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24102025-02-13T05:07:15.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
24112025-02-13T05:07:15.356ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24122025-02-13T05:07:15.356ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
24132025-02-13T05:07:15.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24142025-02-13T05:07:15.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
24152025-02-13T05:07:15.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24162025-02-13T05:07:15.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
24172025-02-13T05:07:15.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24182025-02-13T05:07:15.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
24192025-02-13T05:07:15.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24202025-02-13T05:07:15.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
24212025-02-13T05:07:15.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24222025-02-13T05:07:15.403ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
24232025-02-13T05:07:15.412ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24242025-02-13T05:07:15.412ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
24252025-02-13T05:07:15.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24262025-02-13T05:07:15.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
24272025-02-13T05:07:15.431ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
24282025-02-13T05:07:15.432ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
24292025-02-13T05:07:15.432ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
24302025-02-13T05:07:15.432ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
24312025-02-13T05:07:15.432ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
24322025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
24332025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
24342025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
24352025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
24362025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
24372025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
24382025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
24392025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
24402025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
24412025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
24422025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
24432025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
24442025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
24452025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
24462025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
24472025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
24482025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
24492025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
24502025-02-13T05:07:15.531ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
24512025-02-13T05:07:15.531ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 299914 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = e9c6ed74-d2aa-4129-bda6-3282cdbd6d3b response_code = 503 uri = /omicron-zones
24522025-02-13T05:07:15.531ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e9c6ed74-d2aa-4129-bda6-3282cdbd6d3b", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:15 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "e9c6ed74-d2aa-4129-bda6-3282cdbd6d3b" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 2.817083796s sled_address = [fd00:1122:3344:101::1]:12345
24532025-02-13T05:07:18.342ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
24542025-02-13T05:07:18.342ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
24552025-02-13T05:07:18.342ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
24562025-02-13T05:07:18.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
24572025-02-13T05:07:18.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24582025-02-13T05:07:18.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
24592025-02-13T05:07:18.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24602025-02-13T05:07:18.377ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
24612025-02-13T05:07:18.377ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
24622025-02-13T05:07:18.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
24632025-02-13T05:07:18.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
24642025-02-13T05:07:18.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
24652025-02-13T05:07:18.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
24662025-02-13T05:07:18.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
24672025-02-13T05:07:18.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
24682025-02-13T05:07:18.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
24692025-02-13T05:07:18.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
24702025-02-13T05:07:18.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
24712025-02-13T05:07:18.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24722025-02-13T05:07:18.437ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
24732025-02-13T05:07:18.437ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24742025-02-13T05:07:18.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
24752025-02-13T05:07:18.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24762025-02-13T05:07:18.457ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24772025-02-13T05:07:18.457ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
24782025-02-13T05:07:18.466ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24792025-02-13T05:07:18.466ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
24802025-02-13T05:07:18.476ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24812025-02-13T05:07:18.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
24822025-02-13T05:07:18.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24832025-02-13T05:07:18.486ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
24842025-02-13T05:07:18.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24852025-02-13T05:07:18.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
24862025-02-13T05:07:18.506ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24872025-02-13T05:07:18.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
24882025-02-13T05:07:18.515ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24892025-02-13T05:07:18.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
24902025-02-13T05:07:18.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24912025-02-13T05:07:18.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
24922025-02-13T05:07:18.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
24932025-02-13T05:07:18.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
24942025-02-13T05:07:18.546ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
24952025-02-13T05:07:18.546ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
24962025-02-13T05:07:18.546ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
24972025-02-13T05:07:18.546ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
24982025-02-13T05:07:18.546ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
24992025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
25002025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
25012025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
25022025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
25032025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
25042025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
25052025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
25062025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
25072025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
25082025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
25092025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
25102025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
25112025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
25122025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
25132025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
25142025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
25152025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
25162025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
25172025-02-13T05:07:18.642ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
25182025-02-13T05:07:18.642ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 300873 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = fd151457-9563-4434-a322-5d3c6ab5af09 response_code = 503 uri = /omicron-zones
25192025-02-13T05:07:18.642ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "fd151457-9563-4434-a322-5d3c6ab5af09", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:18 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "fd151457-9563-4434-a322-5d3c6ab5af09" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 5.561895057s sled_address = [fd00:1122:3344:101::1]:12345
25202025-02-13T05:07:24.190ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
25212025-02-13T05:07:24.190ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25222025-02-13T05:07:24.190ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
25232025-02-13T05:07:24.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25242025-02-13T05:07:24.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25252025-02-13T05:07:24.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25262025-02-13T05:07:24.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25272025-02-13T05:07:24.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
25282025-02-13T05:07:24.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
25292025-02-13T05:07:24.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
25302025-02-13T05:07:24.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
25312025-02-13T05:07:24.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
25322025-02-13T05:07:24.241ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
25332025-02-13T05:07:24.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
25342025-02-13T05:07:24.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
25352025-02-13T05:07:24.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
25362025-02-13T05:07:24.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
25372025-02-13T05:07:24.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25382025-02-13T05:07:24.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25392025-02-13T05:07:24.279ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25402025-02-13T05:07:24.282ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25412025-02-13T05:07:24.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
25422025-02-13T05:07:24.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
25432025-02-13T05:07:24.299ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25442025-02-13T05:07:24.301ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
25452025-02-13T05:07:24.308ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25462025-02-13T05:07:24.308ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
25472025-02-13T05:07:24.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25482025-02-13T05:07:24.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
25492025-02-13T05:07:24.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25502025-02-13T05:07:24.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
25512025-02-13T05:07:24.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25522025-02-13T05:07:24.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
25532025-02-13T05:07:24.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25542025-02-13T05:07:24.350ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
25552025-02-13T05:07:24.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25562025-02-13T05:07:24.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
25572025-02-13T05:07:24.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25582025-02-13T05:07:24.369ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
25592025-02-13T05:07:24.378ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
25602025-02-13T05:07:24.378ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
25612025-02-13T05:07:24.388ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
25622025-02-13T05:07:24.388ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
25632025-02-13T05:07:24.388ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
25642025-02-13T05:07:24.388ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
25652025-02-13T05:07:24.388ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
25662025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
25672025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
25682025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
25692025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
25702025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
25712025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
25722025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
25732025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
25742025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
25752025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
25762025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
25772025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
25782025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
25792025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
25802025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
25812025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
25822025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
25832025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
25842025-02-13T05:07:24.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
25852025-02-13T05:07:24.482ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 292895 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 57d65e6c-77a4-4743-bf8d-139e5748ef0a response_code = 503 uri = /omicron-zones
25862025-02-13T05:07:24.482ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "57d65e6c-77a4-4743-bf8d-139e5748ef0a", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:24 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "57d65e6c-77a4-4743-bf8d-139e5748ef0a" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 5.751011381s sled_address = [fd00:1122:3344:101::1]:12345
25872025-02-13T05:07:24.564ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58122
25882025-02-13T05:07:24.564ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 26 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58122 req_id = 666cf5c7-d25d-4d57-a420-7ce239f8f958 response_code = 200 uri = /baseboard
25892025-02-13T05:07:25.886ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 5.974752075s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
25902025-02-13T05:07:30.218ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
25912025-02-13T05:07:30.219ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25922025-02-13T05:07:30.219ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
25932025-02-13T05:07:30.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25942025-02-13T05:07:30.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25952025-02-13T05:07:30.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
25962025-02-13T05:07:30.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25972025-02-13T05:07:30.252ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
25982025-02-13T05:07:30.252ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
25992025-02-13T05:07:30.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26002025-02-13T05:07:30.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
26012025-02-13T05:07:30.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26022025-02-13T05:07:30.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
26032025-02-13T05:07:30.280ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26042025-02-13T05:07:30.280ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
26052025-02-13T05:07:30.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26062025-02-13T05:07:30.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
26072025-02-13T05:07:30.299ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
26082025-02-13T05:07:30.299ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26092025-02-13T05:07:30.309ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
26102025-02-13T05:07:30.309ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26112025-02-13T05:07:30.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
26122025-02-13T05:07:30.320ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26132025-02-13T05:07:30.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26142025-02-13T05:07:30.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
26152025-02-13T05:07:30.342ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26162025-02-13T05:07:30.342ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
26172025-02-13T05:07:30.351ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26182025-02-13T05:07:30.351ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
26192025-02-13T05:07:30.361ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26202025-02-13T05:07:30.361ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
26212025-02-13T05:07:30.370ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26222025-02-13T05:07:30.370ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
26232025-02-13T05:07:30.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26242025-02-13T05:07:30.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
26252025-02-13T05:07:30.390ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26262025-02-13T05:07:30.390ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
26272025-02-13T05:07:30.399ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26282025-02-13T05:07:30.399ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
26292025-02-13T05:07:30.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26302025-02-13T05:07:30.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
26312025-02-13T05:07:30.419ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
26322025-02-13T05:07:30.419ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
26332025-02-13T05:07:30.419ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
26342025-02-13T05:07:30.420ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
26352025-02-13T05:07:30.420ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
26362025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
26372025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
26382025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
26392025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
26402025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
26412025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
26422025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
26432025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
26442025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
26452025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
26462025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
26472025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
26482025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
26492025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
26502025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
26512025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
26522025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
26532025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
26542025-02-13T05:07:30.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
26552025-02-13T05:07:30.513ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 295437 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 903c040b-1408-4737-ab85-57aafe43280a response_code = 503 uri = /omicron-zones
26562025-02-13T05:07:30.513ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "903c040b-1408-4737-ab85-57aafe43280a", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:30 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "903c040b-1408-4737-ab85-57aafe43280a" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 3.951148202s sled_address = [fd00:1122:3344:101::1]:12345
26572025-02-13T05:07:34.455ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
26582025-02-13T05:07:34.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
26592025-02-13T05:07:34.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
26602025-02-13T05:07:34.468ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
26612025-02-13T05:07:34.468ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26622025-02-13T05:07:34.478ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
26632025-02-13T05:07:34.478ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26642025-02-13T05:07:34.489ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26652025-02-13T05:07:34.489ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
26662025-02-13T05:07:34.499ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26672025-02-13T05:07:34.499ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
26682025-02-13T05:07:34.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26692025-02-13T05:07:34.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
26702025-02-13T05:07:34.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26712025-02-13T05:07:34.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
26722025-02-13T05:07:34.529ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
26732025-02-13T05:07:34.529ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
26742025-02-13T05:07:34.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
26752025-02-13T05:07:34.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26762025-02-13T05:07:34.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
26772025-02-13T05:07:34.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26782025-02-13T05:07:34.557ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
26792025-02-13T05:07:34.557ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26802025-02-13T05:07:34.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26812025-02-13T05:07:34.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
26822025-02-13T05:07:34.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26832025-02-13T05:07:34.575ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
26842025-02-13T05:07:34.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26852025-02-13T05:07:34.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
26862025-02-13T05:07:34.593ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26872025-02-13T05:07:34.593ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
26882025-02-13T05:07:34.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26892025-02-13T05:07:34.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
26902025-02-13T05:07:34.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26912025-02-13T05:07:34.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
26922025-02-13T05:07:34.623ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26932025-02-13T05:07:34.623ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
26942025-02-13T05:07:34.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26952025-02-13T05:07:34.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
26962025-02-13T05:07:34.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
26972025-02-13T05:07:34.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
26982025-02-13T05:07:34.651ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
26992025-02-13T05:07:34.651ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
27002025-02-13T05:07:34.651ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
27012025-02-13T05:07:34.651ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
27022025-02-13T05:07:34.651ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27032025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
27042025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
27052025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
27062025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
27072025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
27082025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
27092025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
27102025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
27112025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
27122025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
27132025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
27142025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
27152025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
27162025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
27172025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
27182025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
27192025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
27202025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
27212025-02-13T05:07:34.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3382 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
27222025-02-13T05:07:34.748ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Service Unavailable error_message_internal = Time not yet synchronized file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:855 latency_us = 294127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 0b83d50b-f5c8-497c-84c8-10434f186603 response_code = 503 uri = /omicron-zones
27232025-02-13T05:07:34.749ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "0b83d50b-f5c8-497c-84c8-10434f186603", "content-length": "133", "date": "Thu, 13 Feb 2025 05:07:34 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "0b83d50b-f5c8-497c-84c8-10434f186603" } file = sled-agent/src/rack_setup/service.rs:578 retry_after = 9.581056146s sled_address = [fd00:1122:3344:101::1]:12345
27242025-02-13T05:07:44.305ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 9f4eb8b3-c464-433c-b912-c7432c124258 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc24f910-ffb5-4de8-b7e1-4b9d4d15a7cd (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: d80d65e7-849e-47a7-8d7e-e69f7cc544bb (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 133265e1-ac44-46da-8aab-7104695c8e37 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f945d831-706b-4715-a6b5-1c91eb6084b0, kind: Service { id: 133265e1-ac44-46da-8aab-7104695c8e37 }, name: Name("ntp-133265e1-ac44-46da-8aab-7104695c8e37"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f2470d0d-da0c-4e4f-a170-b6766c01e168 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b13677c-db2d-419d-8efe-35f1b21ab4b8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 40ba9488-97a0-4c82-8eaa-572a3d05f4a3 (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3bd70f31-983e-48d1-b8d7-426a36f6d46a (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 4f62b307-7a1b-4e58-b809-072df20f86f4 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: ec95727a-7af6-45d1-9962-81bf544db84b (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a8269f60-9104-4275-9edc-a31614d5c23e (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f6817c28-f622-48d8-9425-69a8b0c4c59a (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 93aa177d-6d80-4ea9-988c-9de41b46fd8d (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 86121030-8029-4691-8488-13a48de3365f (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: dfa1cf9b-80b9-4548-98f9-0b7785e88a55 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 0ad78864-a307-4845-a8c8-7cbcee9e2ffd (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 9c757498-3ed8-4def-aa91-4a3b71c5bc8d (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 98b8a1c8-74a9-47a3-9522-072db4418496 (service), filesystem_pool: Some(ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External } } } }, OmicronZoneConfig { id: 0de1cab5-0523-4015-b40e-5b41e5aa5909 (service), filesystem_pool: Some(ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External } } } }, OmicronZoneConfig { id: 993b390d-0f46-4377-989b-e802d7fbcf8f (service), filesystem_pool: Some(ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External } } } }, OmicronZoneConfig { id: a94fadfa-137d-4a55-90a2-11fb9b3a2d2e (service), filesystem_pool: Some(ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External } } } }, OmicronZoneConfig { id: 15e07287-8ea0-4d73-ac30-b46735163109 (service), filesystem_pool: Some(ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External } } } }, OmicronZoneConfig { id: 916e23d7-e644-4e84-bf54-e3c395fecfc6 (service), filesystem_pool: Some(ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External } } } }, OmicronZoneConfig { id: c70458e1-909a-4316-9e42-fb91d39aaf86 (service), filesystem_pool: Some(ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb (service), filesystem_pool: Some(ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External } } } }, OmicronZoneConfig { id: e154a1b0-09c7-427d-96ee-87182bf42d9b (service), filesystem_pool: Some(ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
27252025-02-13T05:07:44.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
27262025-02-13T05:07:44.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: InternalDns }, responder: ... }
27272025-02-13T05:07:44.317ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
27282025-02-13T05:07:44.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27292025-02-13T05:07:44.327ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
27302025-02-13T05:07:44.327ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27312025-02-13T05:07:44.337ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
27322025-02-13T05:07:44.337ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Cockroach }, responder: ... }
27332025-02-13T05:07:44.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
27342025-02-13T05:07:44.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Cockroach }, responder: ... }
27352025-02-13T05:07:44.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
27362025-02-13T05:07:44.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
27372025-02-13T05:07:44.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
27382025-02-13T05:07:44.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Cockroach }, responder: ... }
27392025-02-13T05:07:44.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:607
27402025-02-13T05:07:44.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Cockroach }, responder: ... }
27412025-02-13T05:07:44.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
27422025-02-13T05:07:44.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27432025-02-13T05:07:44.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
27442025-02-13T05:07:44.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27452025-02-13T05:07:44.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:607
27462025-02-13T05:07:44.403ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Clickhouse }, responder: ... }
27472025-02-13T05:07:44.411ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27482025-02-13T05:07:44.412ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, kind: Crucible }, responder: ... }
27492025-02-13T05:07:44.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27502025-02-13T05:07:44.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, kind: Crucible }, responder: ... }
27512025-02-13T05:07:44.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27522025-02-13T05:07:44.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, kind: Crucible }, responder: ... }
27532025-02-13T05:07:44.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27542025-02-13T05:07:44.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, kind: Crucible }, responder: ... }
27552025-02-13T05:07:44.450ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27562025-02-13T05:07:44.450ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, kind: Crucible }, responder: ... }
27572025-02-13T05:07:44.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27582025-02-13T05:07:44.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, kind: Crucible }, responder: ... }
27592025-02-13T05:07:44.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27602025-02-13T05:07:44.472ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, kind: Crucible }, responder: ... }
27612025-02-13T05:07:44.479ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27622025-02-13T05:07:44.479ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, kind: Crucible }, responder: ... }
27632025-02-13T05:07:44.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:607
27642025-02-13T05:07:44.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, kind: Crucible }, responder: ... }
27652025-02-13T05:07:44.498ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27662025-02-13T05:07:44.498ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
27672025-02-13T05:07:44.498ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
27682025-02-13T05:07:44.498ZINFOSledAgent (ServiceManager): ensure_all_omicron_zones: request reconciliation done file = sled-agent/src/services.rs:4919 num_zones_to_be_added = 19 num_zones_to_be_removed = 0 num_zones_updated = 0
27692025-02-13T05:07:44.498ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27702025-02-13T05:07:44.610ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
27712025-02-13T05:07:44.611ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27722025-02-13T05:07:44.611ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27732025-02-13T05:07:44.668ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
27742025-02-13T05:07:44.669ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27752025-02-13T05:07:44.669ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27762025-02-13T05:07:44.669ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27772025-02-13T05:07:44.669ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27782025-02-13T05:07:44.669ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27792025-02-13T05:07:44.669ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27802025-02-13T05:07:44.817ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
27812025-02-13T05:07:44.842ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
27822025-02-13T05:07:44.842ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27832025-02-13T05:07:44.842ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27842025-02-13T05:07:44.843ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
27852025-02-13T05:07:44.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27862025-02-13T05:07:44.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27872025-02-13T05:07:44.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27882025-02-13T05:07:44.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
27892025-02-13T05:07:44.879ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f file = illumos-utils/src/zone.rs:325
27902025-02-13T05:07:44.883ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b file = illumos-utils/src/zone.rs:325
27912025-02-13T05:07:44.887ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d file = illumos-utils/src/zone.rs:325
27922025-02-13T05:07:44.892ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e file = illumos-utils/src/zone.rs:325
27932025-02-13T05:07:44.897ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86 file = illumos-utils/src/zone.rs:325
27942025-02-13T05:07:44.902ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f file = illumos-utils/src/zone.rs:325
27952025-02-13T05:07:44.907ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55 file = illumos-utils/src/zone.rs:325
27962025-02-13T05:07:44.912ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb file = illumos-utils/src/zone.rs:325
27972025-02-13T05:07:45.031ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f file = illumos-utils/src/zone.rs:363
27982025-02-13T05:07:45.036ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b file = illumos-utils/src/zone.rs:363
27992025-02-13T05:07:45.041ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d file = illumos-utils/src/zone.rs:363
28002025-02-13T05:07:45.045ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e file = illumos-utils/src/zone.rs:363
28012025-02-13T05:07:45.050ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86 file = illumos-utils/src/zone.rs:363
28022025-02-13T05:07:45.054ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f file = illumos-utils/src/zone.rs:363
28032025-02-13T05:07:45.059ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55 file = illumos-utils/src/zone.rs:363
28042025-02-13T05:07:45.065ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb file = illumos-utils/src/zone.rs:363
28052025-02-13T05:07:45.070ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd file = illumos-utils/src/zone.rs:325
28062025-02-13T05:07:45.075ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496 file = illumos-utils/src/zone.rs:325
28072025-02-13T05:07:45.081ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109 file = illumos-utils/src/zone.rs:325
28082025-02-13T05:07:45.087ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6 file = illumos-utils/src/zone.rs:325
28092025-02-13T05:07:45.094ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d file = illumos-utils/src/zone.rs:325
28102025-02-13T05:07:45.099ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909 file = illumos-utils/src/zone.rs:325
28112025-02-13T05:07:45.163ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
28122025-02-13T05:07:45.163ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd file = illumos-utils/src/zone.rs:363
28132025-02-13T05:07:45.184ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496 file = illumos-utils/src/zone.rs:363
28142025-02-13T05:07:45.211ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109 file = illumos-utils/src/zone.rs:363
28152025-02-13T05:07:45.227ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6 file = illumos-utils/src/zone.rs:363
28162025-02-13T05:07:45.248ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d file = illumos-utils/src/zone.rs:363
28172025-02-13T05:07:45.263ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909 file = illumos-utils/src/zone.rs:363
28182025-02-13T05:07:45.342ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
28192025-02-13T05:07:45.411ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
28202025-02-13T05:07:45.482ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
28212025-02-13T05:07:45.553ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
28222025-02-13T05:07:45.554ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
28232025-02-13T05:07:45.554ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
28242025-02-13T05:07:45.606ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 137, 123]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
28252025-02-13T05:07:45.606ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1321 zone_type = nexus
28262025-02-13T05:07:45.608ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
28272025-02-13T05:07:45.608ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
28282025-02-13T05:07:45.661ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 162, 76]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
28292025-02-13T05:07:45.661ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1321 zone_type = external_dns
28302025-02-13T05:07:45.663ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
28312025-02-13T05:07:45.663ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
28322025-02-13T05:07:45.721ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 131, 247]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
28332025-02-13T05:07:45.721ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1321 zone_type = external_dns
28342025-02-13T05:07:45.722ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
28352025-02-13T05:07:45.722ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
28362025-02-13T05:07:45.779ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 161, 64]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
28372025-02-13T05:07:45.779ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1321 zone_type = nexus
28382025-02-13T05:07:45.779ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
28392025-02-13T05:07:45.779ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
28402025-02-13T05:07:45.779ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
28412025-02-13T05:07:45.779ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
28422025-02-13T05:07:45.781ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
28432025-02-13T05:07:45.781ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
28442025-02-13T05:07:45.837ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 227, 143]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
28452025-02-13T05:07:45.837ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1321 zone_type = nexus
28462025-02-13T05:07:45.837ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
28472025-02-13T05:07:45.839ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
28482025-02-13T05:07:45.841ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
28492025-02-13T05:07:45.841ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
28502025-02-13T05:07:45.995ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
28512025-02-13T05:07:45.996ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
28522025-02-13T05:07:45.996ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
28532025-02-13T05:07:45.996ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
28542025-02-13T05:07:45.996ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228 file = illumos-utils/src/zone.rs:325
28552025-02-13T05:07:46.010ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
28562025-02-13T05:07:46.010ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
28572025-02-13T05:07:46.010ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
28582025-02-13T05:07:46.013ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
28592025-02-13T05:07:46.015ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
28602025-02-13T05:07:46.015ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
28612025-02-13T05:07:46.016ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
28622025-02-13T05:07:46.115ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
28632025-02-13T05:07:46.115ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228 file = illumos-utils/src/zone.rs:363
28642025-02-13T05:07:46.125ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b file = illumos-utils/src/zone.rs:325
28652025-02-13T05:07:46.173ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8 file = illumos-utils/src/zone.rs:325
28662025-02-13T05:07:46.182ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a file = illumos-utils/src/zone.rs:325
28672025-02-13T05:07:46.192ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b file = illumos-utils/src/zone.rs:363
28682025-02-13T05:07:46.208ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e file = illumos-utils/src/zone.rs:325
28692025-02-13T05:07:46.222ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8 file = illumos-utils/src/zone.rs:363
28702025-02-13T05:07:46.232ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a file = illumos-utils/src/zone.rs:363
28712025-02-13T05:07:46.259ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e file = illumos-utils/src/zone.rs:363
28722025-02-13T05:07:46.920ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.303224302s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
28732025-02-13T05:07:53.065ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='14c4b017-6cf3-4ace-9efd-3a0f48b95870'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28742025-02-13T05:07:53.066ZINFOSledAgent (ServiceManager): Booting oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
28752025-02-13T05:07:53.105ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_15d1957c-72d0-47b9-aa2b-993d1474e767/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='cc9022d3-4f66-429b-b91f-4483f1a00aa3'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28762025-02-13T05:07:53.110ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_260eeacc-a97f-47c0-b604-100bb7d05b36/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d8a3a02b-1982-412b-9c6f-46019c9f6f09'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28772025-02-13T05:07:53.111ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='b0fb3588-6328-4c82-b7a0-11a3caa1ff5a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28782025-02-13T05:07:53.111ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='bdd1dc28-2d51-46bf-b148-86b0a8a9c820'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28792025-02-13T05:07:53.112ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_888b0ace-3028-4fa4-a464-87268f90ca4e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d70d99da-3889-449d-9bd1-5afa06ccd33e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28802025-02-13T05:07:53.112ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_92d599bb-7067-4926-9301-d5f5f983f111/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='35b5a5f3-73a0-467e-9fae-c14095ff4131'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28812025-02-13T05:07:53.113ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_9488b08f-08d1-4217-90af-27a0f69b51f7/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5e8468ca-ef24-4e02-bbbb-c05b135a8bf9'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28822025-02-13T05:07:53.113ZINFOSledAgent (ServiceManager): Booting oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
28832025-02-13T05:07:53.123ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28842025-02-13T05:07:53.123ZINFOSledAgent (ServiceManager): Booting oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
28852025-02-13T05:07:53.132ZINFOSledAgent (ServiceManager): Booting oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
28862025-02-13T05:07:53.141ZINFOSledAgent (ServiceManager): Booting oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
28872025-02-13T05:07:53.149ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
28882025-02-13T05:07:53.157ZINFOSledAgent (ServiceManager): Booting oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
28892025-02-13T05:07:53.164ZINFOSledAgent (ServiceManager): Booting oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
28902025-02-13T05:07:53.172ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
28912025-02-13T05:07:53.482ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28922025-02-13T05:07:53.483ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
28932025-02-13T05:07:53.494ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28942025-02-13T05:07:53.494ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
28952025-02-13T05:07:53.685ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='93aa177d-6d80-4ea9-988c-9de41b46fd8d'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28962025-02-13T05:07:53.694ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
28972025-02-13T05:07:53.725ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='92da4b2f-18d1-4002-aad5-aff7c3b9ac54'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
28982025-02-13T05:07:53.822ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 64.43666ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
28992025-02-13T05:07:53.822ZINFOSledAgent (ServiceManager): Booting oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
29002025-02-13T05:07:53.907ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 45.402413ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
29012025-02-13T05:07:54.059ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 32.809257ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
29022025-02-13T05:07:54.109ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 123.687356ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
29032025-02-13T05:07:54.145ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 40.795683ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
29042025-02-13T05:07:54.174ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 35.524414ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
29052025-02-13T05:07:54.225ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 69.886981ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
29062025-02-13T05:07:54.242ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 67.030481ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
29072025-02-13T05:07:54.258ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 50.789196ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
29082025-02-13T05:07:54.273ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 61.863251ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
29092025-02-13T05:07:54.339ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 67.661251ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
29102025-02-13T05:07:54.354ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 29.821304ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
29112025-02-13T05:07:54.368ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 104.858049ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
29122025-02-13T05:07:54.384ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 53.681624ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
29132025-02-13T05:07:54.398ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 44.019618ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
29142025-02-13T05:07:54.413ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 101.748783ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
29152025-02-13T05:07:54.427ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 66.144493ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
29162025-02-13T05:07:54.447ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 100.208932ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
29172025-02-13T05:07:54.468ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 61.80541ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
29182025-02-13T05:07:54.500ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 114.999879ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
29192025-02-13T05:07:54.530ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 111.463912ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
29202025-02-13T05:07:54.547ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 55.750003ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
29212025-02-13T05:07:54.569ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 89.611371ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
29222025-02-13T05:07:54.585ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 101.871275ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
29232025-02-13T05:07:54.598ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 122.987835ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
29242025-02-13T05:07:54.613ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 162.058556ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
29252025-02-13T05:07:54.630ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37130
29262025-02-13T05:07:54.637ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1187 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37130 req_id = 5f168883-4ec5-441a-ad2a-02dba2077576 response_code = 200 uri = /baseboard
29272025-02-13T05:07:54.637ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 174.444807ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
29282025-02-13T05:07:54.662ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 112.73709ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
29292025-02-13T05:07:54.676ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 243.388726ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
29302025-02-13T05:07:54.692ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 132.027371ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
29312025-02-13T05:07:54.710ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 215.67808ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
29322025-02-13T05:07:54.737ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 241.628855ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
29332025-02-13T05:07:54.749ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 194.677779ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
29342025-02-13T05:07:54.777ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 260.559081ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
29352025-02-13T05:07:54.789ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 102.161755ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
29362025-02-13T05:07:54.804ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 124.534478ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
29372025-02-13T05:07:54.821ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 286.877934ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
29382025-02-13T05:07:54.837ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 153.441151ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
29392025-02-13T05:07:54.853ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 458.087193ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
29402025-02-13T05:07:54.873ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 229.801022ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
29412025-02-13T05:07:54.888ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 196.792356ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
29422025-02-13T05:07:54.903ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 205.292049ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
29432025-02-13T05:07:54.920ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 259.076727ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
29442025-02-13T05:07:54.933ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 638.793953ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
29452025-02-13T05:07:54.948ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 341.903366ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
29462025-02-13T05:07:55.117ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 598.143512ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
29472025-02-13T05:07:55.176ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 222.363551ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
29482025-02-13T05:07:55.193ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 325.884026ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
29492025-02-13T05:07:55.212ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 416.024735ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
29502025-02-13T05:07:55.231ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 421.714296ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
29512025-02-13T05:07:55.248ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 871.923801ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
29522025-02-13T05:07:55.263ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 260.507609ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
29532025-02-13T05:07:55.279ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 912.073625ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
29542025-02-13T05:07:55.300ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 410.717206ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
29552025-02-13T05:07:55.303ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29562025-02-13T05:07:55.303ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29572025-02-13T05:07:55.318ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 315.655243ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
29582025-02-13T05:07:55.336ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 813.077669ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
29592025-02-13T05:07:55.356ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 998.257254ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
29602025-02-13T05:07:55.356ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
29612025-02-13T05:07:55.367ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
29622025-02-13T05:07:55.418ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.148610507s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
29632025-02-13T05:07:55.534ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 671.100267ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
29642025-02-13T05:07:55.549ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 434.813074ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
29652025-02-13T05:07:55.589ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.307983545s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
29662025-02-13T05:07:55.648ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 943.942525ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
29672025-02-13T05:07:55.666ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 1.096850945s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
29682025-02-13T05:07:55.685ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 839.994649ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
29692025-02-13T05:07:55.701ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 48.103248ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
29702025-02-13T05:07:55.715ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 68.465472ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
29712025-02-13T05:07:55.734ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 936.051426ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
29722025-02-13T05:07:55.751ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 740.708612ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
29732025-02-13T05:07:55.765ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 129.092685ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
29742025-02-13T05:07:55.800ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 141.807378ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
29752025-02-13T05:07:55.909ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 196.843944ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
29762025-02-13T05:07:55.967ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 195.055878ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
29772025-02-13T05:07:55.996ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 653.669047ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
29782025-02-13T05:07:56.124ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 550.607313ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
29792025-02-13T05:07:56.147ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 963.472103ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
29802025-02-13T05:07:56.165ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 1.336792562s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
29812025-02-13T05:07:56.182ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 218.862659ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
29822025-02-13T05:07:56.205ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 508.508682ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
29832025-02-13T05:07:56.219ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 586.553452ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
29842025-02-13T05:07:56.367ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.055364538s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
29852025-02-13T05:07:56.415ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 408.267884ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
29862025-02-13T05:07:56.504ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 623.410596ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
29872025-02-13T05:07:56.535ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 634.520637ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
29882025-02-13T05:07:56.576ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.197084264s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
29892025-02-13T05:07:56.603ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.079146777s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
29902025-02-13T05:07:56.660ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 1.332345771s file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
29912025-02-13T05:07:56.682ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 594.706703ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
29922025-02-13T05:07:56.696ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 472.092713ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
29932025-02-13T05:07:56.725ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 752.886801ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
29942025-02-13T05:07:56.773ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 1.220799437s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
29952025-02-13T05:07:56.817ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.496206707s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
29962025-02-13T05:07:56.834ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 509.38798ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
29972025-02-13T05:07:56.905ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.498939751s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
29982025-02-13T05:07:57.123ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 578.179968ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
29992025-02-13T05:07:57.141ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 960.781908ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
30002025-02-13T05:07:57.182ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.328789326s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
30012025-02-13T05:07:57.193ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 636.131319ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
30022025-02-13T05:07:57.288ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 748.918741ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
30032025-02-13T05:07:57.356ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.133969914s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
30042025-02-13T05:07:57.431ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 985.801569ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
30052025-02-13T05:07:57.487ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 922.921838ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
30062025-02-13T05:07:57.509ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 1.168088422s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
30072025-02-13T05:07:57.692ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 684.652569ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
30082025-02-13T05:07:57.713ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 914.041181ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
30092025-02-13T05:07:57.782ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 853.540858ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
30102025-02-13T05:07:57.811ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30112025-02-13T05:07:57.811ZINFOSledAgent (ServiceManager): Booting oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30122025-02-13T05:07:57.850ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 1.081641086s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
30132025-02-13T05:07:57.863ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30142025-02-13T05:07:57.863ZINFOSledAgent (ServiceManager): Booting oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228 zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30152025-02-13T05:07:58.001ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 746.757481ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
30162025-02-13T05:07:58.015ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 937.234791ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
30172025-02-13T05:07:58.034ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 25.467036ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30182025-02-13T05:07:58.045ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 779.106046ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
30192025-02-13T05:07:58.067ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 59.374865ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30202025-02-13T05:07:58.077ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 133.775865ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30212025-02-13T05:07:58.110ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 752.757243ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
30222025-02-13T05:07:58.137ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 78.940693ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30232025-02-13T05:07:58.222ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 289.643545ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30242025-02-13T05:07:58.231ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 199.059388ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30252025-02-13T05:07:58.323ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.038175269s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
30262025-02-13T05:07:58.391ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.188470956s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
30272025-02-13T05:07:58.419ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.393414939s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
30282025-02-13T05:07:58.438ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 831.741363ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
30292025-02-13T05:07:58.455ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 515.844367ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
30302025-02-13T05:07:58.475ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 493.338761ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30312025-02-13T05:07:58.503ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 889.481841ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
30322025-02-13T05:07:58.523ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 948.27489ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
30332025-02-13T05:07:58.541ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 355.631271ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30342025-02-13T05:07:58.638ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 627.419314ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
30352025-02-13T05:07:58.649ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.488467897s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
30362025-02-13T05:07:58.685ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 781.910856ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
30372025-02-13T05:07:58.762ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 721.577315ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
30382025-02-13T05:07:58.921ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.469808486s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
30392025-02-13T05:07:59.281ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.393632816s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
30402025-02-13T05:07:59.316ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 661.220062ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30412025-02-13T05:07:59.331ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 1.093028251s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
30422025-02-13T05:07:59.345ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 968.279294ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
30432025-02-13T05:07:59.359ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.330443168s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
30442025-02-13T05:07:59.374ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 415.812096ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30452025-02-13T05:07:59.389ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 891.080589ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
30462025-02-13T05:07:59.404ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 929.662615ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
30472025-02-13T05:07:59.421ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 825.0704ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
30482025-02-13T05:07:59.440ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 998.757533ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
30492025-02-13T05:07:59.491ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 1.256188201s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
30502025-02-13T05:07:59.510ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 668.254137ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
30512025-02-13T05:07:59.528ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 922.027424ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
30522025-02-13T05:07:59.594ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 543.029197ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
30532025-02-13T05:07:59.809ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 891.472262ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30542025-02-13T05:07:59.826ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 664.855992ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
30552025-02-13T05:07:59.907ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30562025-02-13T05:07:59.907ZINFOSledAgent (ServiceManager): Booting oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
30572025-02-13T05:07:59.991ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.11830976s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30582025-02-13T05:08:00.147ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 766.649077ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
30592025-02-13T05:08:00.158ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.277667991s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
30602025-02-13T05:08:00.167ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 41.450907ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
30612025-02-13T05:08:00.190ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 877.988728ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
30622025-02-13T05:08:00.222ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 149.546382ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
30632025-02-13T05:08:00.259ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.422756185s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
30642025-02-13T05:08:00.291ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 1.099094803s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
30652025-02-13T05:08:00.325ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 999.485314ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
30662025-02-13T05:08:00.346ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.046749221s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
30672025-02-13T05:08:00.383ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 229.315682ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
30682025-02-13T05:08:00.401ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.482936407s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
30692025-02-13T05:08:00.437ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 1.05927909s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
30702025-02-13T05:08:00.451ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 812.76476ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
30712025-02-13T05:08:00.463ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 827.94174ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
30722025-02-13T05:08:00.503ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 891.609032ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
30732025-02-13T05:08:00.642ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 554.671444ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
30742025-02-13T05:08:00.684ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 982.240057ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
30752025-02-13T05:08:00.697ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 939.19839ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
30762025-02-13T05:08:00.712ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 1.323195969s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
30772025-02-13T05:08:00.758ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 1.297647755s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
30782025-02-13T05:08:00.927ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.343008724s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
30792025-02-13T05:08:01.080ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 949.633781ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
30802025-02-13T05:08:01.119ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.258675537s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
30812025-02-13T05:08:01.209ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 730.026371ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
30822025-02-13T05:08:01.274ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 754.824261ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
30832025-02-13T05:08:01.303ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 918.418945ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
30842025-02-13T05:08:01.337ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 546.233834ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
30852025-02-13T05:08:01.404ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 693.747687ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
30862025-02-13T05:08:01.418ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.260937126s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
30872025-02-13T05:08:01.430ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.179525244s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
30882025-02-13T05:08:01.445ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 666.569127ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
30892025-02-13T05:08:01.508ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 884.714131ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
30902025-02-13T05:08:01.649ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 816.695317ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
30912025-02-13T05:08:01.658ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin-single"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:9000'/> <propval type="astring" name="ch_binary" value='/opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30922025-02-13T05:08:01.659ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f zone file = illumos-utils/src/running_zone.rs:489 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
30932025-02-13T05:08:01.679ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.026170213s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
30942025-02-13T05:08:01.694ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.157796539s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
30952025-02-13T05:08:01.891ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 43.374373ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
30962025-02-13T05:08:01.907ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.059703029s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
30972025-02-13T05:08:01.920ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 868.895554ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
30982025-02-13T05:08:01.947ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 100.937438ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
30992025-02-13T05:08:01.960ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 1.082839521s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
31002025-02-13T05:08:02.052ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 861.332038ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
31012025-02-13T05:08:02.077ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.08228975s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
31022025-02-13T05:08:02.089ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 952.805773ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
31032025-02-13T05:08:02.100ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 1.392129184s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
31042025-02-13T05:08:02.112ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 247.247262ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
31052025-02-13T05:08:02.123ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 1.421730894s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
31062025-02-13T05:08:02.133ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.165462304s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
31072025-02-13T05:08:02.233ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.169204078s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
31082025-02-13T05:08:02.281ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.340087617s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
31092025-02-13T05:08:02.374ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 313.034043ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
31102025-02-13T05:08:02.390ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.320389974s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
31112025-02-13T05:08:02.404ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 770.883586ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
31122025-02-13T05:08:02.480ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.488370002s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
31132025-02-13T05:08:02.623ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.481648614s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
31142025-02-13T05:08:02.693ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 839.402554ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
31152025-02-13T05:08:02.706ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 1.179918905s file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
31162025-02-13T05:08:02.719ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.416107099s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
31172025-02-13T05:08:02.802ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 854.72119ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
31182025-02-13T05:08:02.865ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.206815493s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
31192025-02-13T05:08:02.927ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 1.217841953s file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
31202025-02-13T05:08:02.981ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.300917429s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
31212025-02-13T05:08:03.055ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 905.304137ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
31222025-02-13T05:08:03.067ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 952.014189ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
31232025-02-13T05:08:03.175ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.243241598s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
31242025-02-13T05:08:03.189ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 858.343932ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
31252025-02-13T05:08:03.314ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.207724004s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
31262025-02-13T05:08:03.418ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 750.129033ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
31272025-02-13T05:08:03.509ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 645.543212ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
31282025-02-13T05:08:03.548ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 664.296732ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
31292025-02-13T05:08:03.563ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 643.953279ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
31302025-02-13T05:08:03.635ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.495223869s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
31312025-02-13T05:08:03.672ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 920.66291ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
31322025-02-13T05:08:03.726ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.084610334s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
31332025-02-13T05:08:03.906ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 830.726746ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
31342025-02-13T05:08:03.978ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 1.191230765s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
31352025-02-13T05:08:03.992ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.277271762s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
31362025-02-13T05:08:04.034ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 1.139740805s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
31372025-02-13T05:08:04.062ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 537.434619ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
31382025-02-13T05:08:04.088ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.289902598s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
31392025-02-13T05:08:04.129ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.401718511s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
31402025-02-13T05:08:04.155ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.133232425s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
31412025-02-13T05:08:04.173ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 826.129098ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
31422025-02-13T05:08:04.190ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 1.471211299s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
31432025-02-13T05:08:04.208ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.424086381s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
31442025-02-13T05:08:04.226ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 649.922438ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
31452025-02-13T05:08:04.245ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 630.570287ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
31462025-02-13T05:08:04.325ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.159107435s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
31472025-02-13T05:08:04.444ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 779.36674ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
31482025-02-13T05:08:04.539ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.303739112s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
31492025-02-13T05:08:04.613ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 526.375287ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
31502025-02-13T05:08:04.631ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 519.902892ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
31512025-02-13T05:08:04.753ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 1.23297928s file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
31522025-02-13T05:08:04.840ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 504.547603ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
31532025-02-13T05:08:04.894ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.336566572s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
31542025-02-13T05:08:04.911ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 814.568458ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
31552025-02-13T05:08:05.018ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 752.120255ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
31562025-02-13T05:08:05.147ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.096340855s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
31572025-02-13T05:08:05.161ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 1.44218793s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
31582025-02-13T05:08:05.173ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 1.008203148s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
31592025-02-13T05:08:05.187ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 629.560312ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
31602025-02-13T05:08:05.201ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 1.110573672s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
31612025-02-13T05:08:05.237ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.153659951s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
31622025-02-13T05:08:05.285ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.484924577s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
31632025-02-13T05:08:05.303ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.194194691s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
31642025-02-13T05:08:05.358ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 657.793664ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
31652025-02-13T05:08:05.392ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.400115257s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
31662025-02-13T05:08:05.497ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 564.838408ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
31672025-02-13T05:08:05.545ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 977.009434ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
31682025-02-13T05:08:05.648ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.359392025s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
31692025-02-13T05:08:05.676ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 1.456645523s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
31702025-02-13T05:08:05.738ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 1.339166146s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
31712025-02-13T05:08:05.782ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 519.934703ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
31722025-02-13T05:08:05.831ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 710.732347ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
31732025-02-13T05:08:05.857ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.126811349s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
31742025-02-13T05:08:05.998ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 1.453059394s file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
31752025-02-13T05:08:06.029ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 599.609478ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
31762025-02-13T05:08:06.076ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.414429531s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
31772025-02-13T05:08:06.192ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 645.243054ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
31782025-02-13T05:08:06.242ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.163881932s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
31792025-02-13T05:08:06.257ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.058712275s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
31802025-02-13T05:08:06.315ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 1.17899587s file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
31812025-02-13T05:08:06.326ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 1.470222837s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
31822025-02-13T05:08:06.407ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 802.386007ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
31832025-02-13T05:08:06.512ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 573.489507ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
31842025-02-13T05:08:06.539ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.090818457s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
31852025-02-13T05:08:06.556ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 587.523219ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
31862025-02-13T05:08:06.618ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 710.483909ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
31872025-02-13T05:08:06.642ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 579.567729ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
31882025-02-13T05:08:06.785ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.384141599s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
31892025-02-13T05:08:06.808ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 754.986876ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
31902025-02-13T05:08:06.850ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 1.011109969s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
31912025-02-13T05:08:06.997ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.064306487s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
31922025-02-13T05:08:07.019ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.106284893s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
31932025-02-13T05:08:07.090ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 1.495719449s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
31942025-02-13T05:08:07.101ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 952.825787ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
31952025-02-13T05:08:07.145ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 566.814766ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
31962025-02-13T05:08:07.156ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 950.770284ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
31972025-02-13T05:08:07.221ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.351806654s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
31982025-02-13T05:08:07.233ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.295669036s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
31992025-02-13T05:08:07.327ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.350103397s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
32002025-02-13T05:08:07.338ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 1.424850145s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
32012025-02-13T05:08:07.423ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.264052032s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
32022025-02-13T05:08:07.470ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 768.696311ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
32032025-02-13T05:08:07.503ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 635.619921ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
32042025-02-13T05:08:07.515ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 829.180049ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
32052025-02-13T05:08:07.575ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 849.647487ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
32062025-02-13T05:08:07.642ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 718.842314ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
32072025-02-13T05:08:07.723ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 815.628355ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
32082025-02-13T05:08:07.808ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 1.247213331s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
32092025-02-13T05:08:07.858ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 14.458511489s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
32102025-02-13T05:08:07.874ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 811.799632ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
32112025-02-13T05:08:08.069ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.047862783s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
32122025-02-13T05:08:08.082ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.112078858s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
32132025-02-13T05:08:08.121ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 1.306586807s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
32142025-02-13T05:08:08.139ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.476487751s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
32152025-02-13T05:08:08.153ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.429703717s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
32162025-02-13T05:08:08.185ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.154097552s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
32172025-02-13T05:08:08.251ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 1.305848307s file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
32182025-02-13T05:08:08.356ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 1.153801719s file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
32192025-02-13T05:08:08.375ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 539.478332ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
32202025-02-13T05:08:08.442ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.049929165s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
32212025-02-13T05:08:08.545ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.374376401s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
32222025-02-13T05:08:08.559ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 1.48136168s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
32232025-02-13T05:08:08.584ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.236978957s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
32242025-02-13T05:08:08.597ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 916.130136ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
32252025-02-13T05:08:08.687ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.050568448s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
32262025-02-13T05:08:08.697ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 815.955185ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
32272025-02-13T05:08:08.708ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.063391789s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
32282025-02-13T05:08:08.775ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 545.4517ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
32292025-02-13T05:08:08.930ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.428860857s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
32302025-02-13T05:08:09.069ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 683.295822ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
32312025-02-13T05:08:09.129ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.216563315s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
32322025-02-13T05:08:09.205ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.27433671s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
32332025-02-13T05:08:09.331ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 1.006488158s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
32342025-02-13T05:08:09.349ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.258369007s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
32352025-02-13T05:08:09.437ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 514.435488ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
32362025-02-13T05:08:09.501ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 565.507908ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
32372025-02-13T05:08:09.521ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 1.091439062s file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
32382025-02-13T05:08:09.529ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 809.462146ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
32392025-02-13T05:08:09.537ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 556.647836ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
32402025-02-13T05:08:09.566ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 1.357013328s file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
32412025-02-13T05:08:09.593ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 1.366374366s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
32422025-02-13T05:08:09.625ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 921.393563ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
32432025-02-13T05:08:09.749ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 994.147092ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
32442025-02-13T05:08:09.762ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 1.496393311s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
32452025-02-13T05:08:09.780ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 921.209504ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
32462025-02-13T05:08:09.830ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 666.98173ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
32472025-02-13T05:08:09.929ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.37216568s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
32482025-02-13T05:08:09.961ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 935.824459ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
32492025-02-13T05:08:10.050ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 979.74884ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
32502025-02-13T05:08:10.075ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 898.391798ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
32512025-02-13T05:08:10.104ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 894.559426ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
32522025-02-13T05:08:10.349ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909") failed: Property not found. retry in 1.444029002s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
32532025-02-13T05:08:10.359ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 1.266727667s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
32542025-02-13T05:08:10.367ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 547.122495ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
32552025-02-13T05:08:10.375ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.217328136s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
32562025-02-13T05:08:10.489ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 1.043335183s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
32572025-02-13T05:08:10.507ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6") failed: Property not found. retry in 1.323097943s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
32582025-02-13T05:08:10.556ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.498323665s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
32592025-02-13T05:08:10.617ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 639.704568ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
32602025-02-13T05:08:10.628ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 682.290326ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
32612025-02-13T05:08:10.712ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e") failed: Property not found. retry in 1.371744532s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
32622025-02-13T05:08:10.754ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 1.342739981s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
32632025-02-13T05:08:10.906ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 854.345646ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
32642025-02-13T05:08:10.925ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 995.380913ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
32652025-02-13T05:08:10.943ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 930.251736ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
32662025-02-13T05:08:10.969ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 687.976966ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
32672025-02-13T05:08:10.983ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 559.566092ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
32682025-02-13T05:08:11.007ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 738.062818ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
32692025-02-13T05:08:11.039ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 828.476488ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
32702025-02-13T05:08:11.267ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 1.437792308s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
32712025-02-13T05:08:11.276ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 1.058523367s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
32722025-02-13T05:08:11.310ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 602.607142ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
32732025-02-13T05:08:11.319ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 1.272813484s file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
32742025-02-13T05:08:11.543ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 759.446229ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
32752025-02-13T05:08:11.552ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 741.309522ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
32762025-02-13T05:08:11.601ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 729.566279ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
32772025-02-13T05:08:11.635ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 1.294995872s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
32782025-02-13T05:08:11.667ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 946.070458ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
32792025-02-13T05:08:11.755ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 1.197486945s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
32802025-02-13T05:08:11.770ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 1.021401153s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
32812025-02-13T05:08:11.879ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 704.149317ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
32822025-02-13T05:08:11.888ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 996.095463ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
32832025-02-13T05:08:11.924ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.010210941s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
32842025-02-13T05:08:11.932ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 1.243196706s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
32852025-02-13T05:08:12.066ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.009036032s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
32862025-02-13T05:08:12.109ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 810.707638ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
32872025-02-13T05:08:12.304ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 1.309516945s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
32882025-02-13T05:08:12.312ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d") failed: Property not found. retry in 720.139619ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
32892025-02-13T05:08:12.340ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.11000725s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
32902025-02-13T05:08:12.348ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 999.280826ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
32912025-02-13T05:08:12.594ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228") failed: Property not found. retry in 1.025335892s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
32922025-02-13T05:08:12.603ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 702.583173ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
32932025-02-13T05:08:12.621ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 799.681206ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
32942025-02-13T05:08:12.714ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b") failed: Property not found. retry in 954.247727ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
32952025-02-13T05:08:12.802ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f") failed: Property not found. retry in 687.766549ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
32962025-02-13T05:08:12.894ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 1.118808065s file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
32972025-02-13T05:08:12.928ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496") failed: Property not found. retry in 773.480324ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
32982025-02-13T05:08:12.940ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86") failed: Property not found. retry in 1.368290248s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
32992025-02-13T05:08:12.950ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a") failed: Property not found. retry in 1.237236237s file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
33002025-02-13T05:08:12.962ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109") failed: Property not found. retry in 533.881112ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
33012025-02-13T05:08:13.257ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd") failed: Property not found. retry in 594.322382ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
33022025-02-13T05:08:13.267ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8") failed: Property not found. retry in 1.429308012s file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
33032025-02-13T05:08:13.318ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d") failed: Property not found. retry in 862.297913ms file = illumos-utils/src/svc.rs:42 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
33042025-02-13T05:08:13.357ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e") failed: Property not found. retry in 791.488426ms file = illumos-utils/src/svc.rs:42 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
33052025-02-13T05:08:13.432ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb") failed: Property not found. retry in 632.969748ms file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
33062025-02-13T05:08:13.459ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55") failed: Property not found. retry in 1.462965632s file = illumos-utils/src/svc.rs:42 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
33072025-02-13T05:08:13.629ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b") failed: Property not found. retry in 565.990612ms file = illumos-utils/src/svc.rs:42 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
33082025-02-13T05:08:14.087ZWARNSledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f") failed: Property not found. retry in 966.715013ms file = illumos-utils/src/svc.rs:42 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
33092025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_993b390d-0f46-4377-989b-e802d7fbcf8f
33102025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_e154a1b0-09c7-427d-96ee-87182bf42d9b
33112025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_nexus_67cd1f2c-6dc2-422f-a2c8-74ebf346d228
33122025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_oximeter_93aa177d-6d80-4ea9-988c-9de41b46fd8d
33132025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_a94fadfa-137d-4a55-90a2-11fb9b3a2d2e
33142025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_c70458e1-909a-4316-9e42-fb91d39aaf86
33152025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_clickhouse_86121030-8029-4691-8488-13a48de3365f
33162025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_pantry_dfa1cf9b-80b9-4548-98f9-0b7785e88a55
33172025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_14edd0d7-1ee5-4971-8d3e-b7c3ba6a24fb
33182025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_external_dns_ec95727a-7af6-45d1-9962-81bf544db84b
33192025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_nexus_a8269f60-9104-4275-9edc-a31614d5c23e
33202025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_pantry_0ad78864-a307-4845-a8c8-7cbcee9e2ffd
33212025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_98b8a1c8-74a9-47a3-9522-072db4418496
33222025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_external_dns_f580e8f1-7812-496c-8421-463d5f7ea1f8
33232025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_nexus_f6817c28-f622-48d8-9425-69a8b0c4c59a
33242025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_15e07287-8ea0-4d73-ac30-b46735163109
33252025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_916e23d7-e644-4e84-bf54-e3c395fecfc6
33262025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_pantry_9c757498-3ed8-4def-aa91-4a3b71c5bc8d
33272025-02-13T05:08:15.083ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3378 zone = oxz_crucible_0de1cab5-0523-4015-b40e-5b41e5aa5909
33282025-02-13T05:08:15.085ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33292025-02-13T05:08:15.091ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33302025-02-13T05:08:15.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 30842553 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53368 req_id = 6a3134ad-4175-4369-bcb8-eaf381c30cc8 response_code = 204 uri = /omicron-zones
33312025-02-13T05:08:15.111ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1406
33322025-02-13T05:08:15.481ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:790
33332025-02-13T05:08:15.490ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:803
33342025-02-13T05:08:15.549ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, rib_priority: None, vlan_id: None, }, ], switch: Switch0, tx_eq: None, uplink_port_fec: Some( None, ), uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:1008
33352025-02-13T05:08:19.469ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33362025-02-13T05:08:19.537ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33372025-02-13T05:08:19.630ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33382025-02-13T05:08:19.696ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/nexus.rs:530 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
33392025-02-13T05:08:19.792ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33402025-02-13T05:08:19.960ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33412025-02-13T05:08:20.074ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33422025-02-13T05:08:20.145ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/nexus.rs:530 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
33432025-02-13T05:08:20.350ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33442025-02-13T05:08:20.648ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33452025-02-13T05:08:20.904ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33462025-02-13T05:08:21.294ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33472025-02-13T05:08:21.942ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33482025-02-13T05:08:22.143ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5835b48f-f45d-4fa8-91ed-38d933d73732) file = sled-agent/src/nexus.rs:530 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
33492025-02-13T05:08:22.358ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 22.995865756s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
33502025-02-13T05:08:22.703ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33512025-02-13T05:08:23.320ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33522025-02-13T05:08:23.862ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/c3272b8c-876d-43d9-a1f4-8963a7553b1d/initialization-complete) file = sled-agent/src/rack_setup/service.rs:1070
33532025-02-13T05:08:24.665ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:63355
33542025-02-13T05:08:24.666ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 812 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:63355 req_id = c75ca373-686b-4ad6-9fe5-143303b409d1 response_code = 200 uri = /baseboard
33552025-02-13T05:08:25.580ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "3d6874b4-ff11-47e6-8349-ea1291a25d91", "content-length": "173", "date": "Thu, 13 Feb 2025 05:08:25 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"c3272b8c-876d-43d9-a1f4-8963a7553b1d\"", request_id: "3d6874b4-ff11-47e6-8349-ea1291a25d91" } file = sled-agent/src/rack_setup/service.rs:1070
33562025-02-13T05:08:26.330ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:503 new_generation = 1 old_generation = 1 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
33572025-02-13T05:08:26.748ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50934
33582025-02-13T05:08:26.749ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50934 req_id = 8d0204eb-81f4-43b1-a3f3-40dfed46ed90 response_code = 200 uri = /vpc-routes
33592025-02-13T05:08:26.772ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53916
33602025-02-13T05:08:26.773ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53916 req_id = 1eb450c6-c2b0-4e86-8b35-e93f396bf2c3 response_code = 200 uri = /vpc-routes
33612025-02-13T05:08:26.775ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52209
33622025-02-13T05:08:26.778ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
33632025-02-13T05:08:26.779ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45992
33642025-02-13T05:08:26.779ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38804
33652025-02-13T05:08:26.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38804 req_id = a9e383f5-d47b-413d-99bf-8569fb6f9dbe response_code = 200 uri = /vpc-routes
33662025-02-13T05:08:26.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5405 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52209 req_id = a69e39d8-29d6-4cfe-a1da-f1a24603f4ce response_code = 200 uri = /artifacts
33672025-02-13T05:08:26.781ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
33682025-02-13T05:08:26.781ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
33692025-02-13T05:08:26.781ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
33702025-02-13T05:08:26.784ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
33712025-02-13T05:08:26.802ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
33722025-02-13T05:08:26.802ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
33732025-02-13T05:08:26.813ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39705
33742025-02-13T05:08:26.821ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
33752025-02-13T05:08:26.821ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
33762025-02-13T05:08:26.821ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
33772025-02-13T05:08:26.825ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41364
33782025-02-13T05:08:26.826ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
33792025-02-13T05:08:26.827ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
33802025-02-13T05:08:26.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5705 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41364 req_id = 3d59f856-a5f6-491a-be3e-292c06a01eb7 response_code = 200 uri = /artifacts
33812025-02-13T05:08:26.836ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
33822025-02-13T05:08:26.836ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
33832025-02-13T05:08:26.849ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49321
33842025-02-13T05:08:26.854ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
33852025-02-13T05:08:26.854ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
33862025-02-13T05:08:26.861ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
33872025-02-13T05:08:26.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 2661 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50934 req_id = 296dedaf-5f70-498c-a0a7-de4894d64a35 response_code = 204 uri = /eip-gateways
33882025-02-13T05:08:26.873ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
33892025-02-13T05:08:26.885ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 35383 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49321 req_id = 021b3d12-a919-459a-a4b7-5a0b4be359ac response_code = 200 uri = /artifacts
33902025-02-13T05:08:26.889ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
33912025-02-13T05:08:26.889ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
33922025-02-13T05:08:26.896ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
33932025-02-13T05:08:26.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38804 req_id = 51e641b8-a391-4fb5-bc25-f2dd942f3eb1 response_code = 204 uri = /eip-gateways
33942025-02-13T05:08:26.897ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
33952025-02-13T05:08:26.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 34 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53916 req_id = a9b4cb37-722d-4597-8876-3dfb601ee0f4 response_code = 204 uri = /eip-gateways
33962025-02-13T05:08:26.907ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
33972025-02-13T05:08:26.908ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
33982025-02-13T05:08:26.940ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
33992025-02-13T05:08:26.941ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
34002025-02-13T05:08:26.949ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53420
34012025-02-13T05:08:26.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1746 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53420 req_id = d35d507d-8508-4d16-8a46-93adeb2c9015 response_code = 200 uri = /v2p
34022025-02-13T05:08:26.956ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62172
34032025-02-13T05:08:26.956ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 161 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62172 req_id = 29411399-5f12-4206-9c96-cfc786042ceb response_code = 200 uri = /v2p
34042025-02-13T05:08:26.957ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34052025-02-13T05:08:26.957ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
34062025-02-13T05:08:26.968ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33043
34072025-02-13T05:08:26.974ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
34082025-02-13T05:08:26.974ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
34092025-02-13T05:08:26.974ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
34102025-02-13T05:08:26.978ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34112025-02-13T05:08:26.979ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
34122025-02-13T05:08:26.985ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
34132025-02-13T05:08:26.998ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34142025-02-13T05:08:26.998ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
34152025-02-13T05:08:27.013ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34162025-02-13T05:08:27.013ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
34172025-02-13T05:08:27.030ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34182025-02-13T05:08:27.031ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
34192025-02-13T05:08:27.052ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34202025-02-13T05:08:27.052ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
34212025-02-13T05:08:27.066ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5521 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50934 req_id = 61c71538-70c1-446c-8034-421bfa1289f3 response_code = 204 uri = /vpc-routes
34222025-02-13T05:08:27.073ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34232025-02-13T05:08:27.073ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
34242025-02-13T05:08:27.091ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34252025-02-13T05:08:27.092ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
34262025-02-13T05:08:27.097ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41535
34272025-02-13T05:08:27.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 308 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41535 req_id = 4cb9825a-dbd7-4ee0-9594-d504eb357098 response_code = 200 uri = /v2p
34282025-02-13T05:08:27.108ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34292025-02-13T05:08:27.108ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
34302025-02-13T05:08:27.124ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34312025-02-13T05:08:27.124ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
34322025-02-13T05:08:27.135ZINFOSledAgent (PortManager): skipping delta compute for subnet file = illumos-utils/src/opte/port_manager.rs:486 new_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 4 } old_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 4 } subnet = RouterId { vni: Vni(100), kind: System }
34332025-02-13T05:08:27.135ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 581 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38804 req_id = a6fc9eb0-ba69-4cdd-837d-053d0c85ec48 response_code = 204 uri = /vpc-routes
34342025-02-13T05:08:27.142ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34352025-02-13T05:08:27.142ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
34362025-02-13T05:08:27.159ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34372025-02-13T05:08:27.160ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
34382025-02-13T05:08:27.169ZINFOSledAgent (PortManager): skipping delta compute for subnet file = illumos-utils/src/opte/port_manager.rs:486 new_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 4 } old_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 4 } subnet = RouterId { vni: Vni(100), kind: System }
34392025-02-13T05:08:27.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5113 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53916 req_id = b2d266f0-4b85-43e6-9fa1-a949039e0ada response_code = 204 uri = /vpc-routes
34402025-02-13T05:08:27.177ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34412025-02-13T05:08:27.177ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
34422025-02-13T05:08:27.192ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34432025-02-13T05:08:27.192ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
34442025-02-13T05:08:27.208ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34452025-02-13T05:08:27.208ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
34462025-02-13T05:08:27.222ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34472025-02-13T05:08:27.226ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
34482025-02-13T05:08:27.241ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34492025-02-13T05:08:27.241ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
34502025-02-13T05:08:27.256ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34512025-02-13T05:08:27.256ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
34522025-02-13T05:08:27.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 445864 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39705 req_id = 68458d96-c825-4a95-ba20-97749ee2f59a response_code = 200 uri = /inventory
34532025-02-13T05:08:27.274ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 494808 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45992 req_id = dd869576-2810-42f6-bd27-e601dfe7314b response_code = 200 uri = /inventory
34542025-02-13T05:08:27.280ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34552025-02-13T05:08:27.280ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
34562025-02-13T05:08:27.304ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34572025-02-13T05:08:27.304ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
34582025-02-13T05:08:27.328ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34592025-02-13T05:08:27.328ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
34602025-02-13T05:08:27.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 373594 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33043 req_id = e1a1c634-23f3-4456-81c2-05d7c740e566 response_code = 200 uri = /inventory
34612025-02-13T05:08:28.079ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60095
34622025-02-13T05:08:28.079ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
34632025-02-13T05:08:28.079ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
34642025-02-13T05:08:28.080ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
34652025-02-13T05:08:28.080ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
34662025-02-13T05:08:28.092ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34672025-02-13T05:08:28.092ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
34682025-02-13T05:08:28.116ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34692025-02-13T05:08:28.116ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
34702025-02-13T05:08:28.140ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34712025-02-13T05:08:28.140ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
34722025-02-13T05:08:28.160ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34732025-02-13T05:08:28.161ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
34742025-02-13T05:08:28.182ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34752025-02-13T05:08:28.182ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
34762025-02-13T05:08:28.202ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34772025-02-13T05:08:28.202ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
34782025-02-13T05:08:28.222ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34792025-02-13T05:08:28.222ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
34802025-02-13T05:08:28.242ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34812025-02-13T05:08:28.243ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
34822025-02-13T05:08:28.264ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
34832025-02-13T05:08:28.264ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
34842025-02-13T05:08:28.277ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 197132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60095 req_id = fcdcffd9-e123-44d7-beb1-81a9d139074d response_code = 200 uri = /inventory
34852025-02-13T05:08:28.305ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64607
34862025-02-13T05:08:28.307ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34872025-02-13T05:08:28.307ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34882025-02-13T05:08:28.307ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34892025-02-13T05:08:28.307ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34902025-02-13T05:08:28.307ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34912025-02-13T05:08:28.307ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
34922025-02-13T05:08:28.308ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34932025-02-13T05:08:28.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 2252 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64607 req_id = db15b4ed-e1b0-4d00-9d16-9fb1400dc8ba response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34942025-02-13T05:08:28.308ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:1080
34952025-02-13T05:08:28.311ZINFOSledAgent (RSS): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
34962025-02-13T05:08:28.311ZINFOSledAgent (RSS): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
34972025-02-13T05:08:28.807ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33345
34982025-02-13T05:08:28.808ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34992025-02-13T05:08:28.808ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35002025-02-13T05:08:28.808ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35012025-02-13T05:08:28.808ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35022025-02-13T05:08:28.808ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
35032025-02-13T05:08:28.808ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
35042025-02-13T05:08:28.808ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
35052025-02-13T05:08:28.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 348 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33345 req_id = 6de057ad-729d-41c4-a26b-641d12db7725 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
35062025-02-13T05:08:28.988ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36023
35072025-02-13T05:08:28.988ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
35082025-02-13T05:08:28.988ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
35092025-02-13T05:08:28.988ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
35102025-02-13T05:08:28.988ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
35112025-02-13T05:08:29.002ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35122025-02-13T05:08:29.002ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
35132025-02-13T05:08:29.029ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35142025-02-13T05:08:29.029ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
35152025-02-13T05:08:29.056ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35162025-02-13T05:08:29.056ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
35172025-02-13T05:08:29.070ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37748
35182025-02-13T05:08:29.070ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
35192025-02-13T05:08:29.070ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35202025-02-13T05:08:29.070ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35212025-02-13T05:08:29.070ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35222025-02-13T05:08:29.070ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
35232025-02-13T05:08:29.070ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
35242025-02-13T05:08:29.070ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
35252025-02-13T05:08:29.070ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 506 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37748 req_id = 9e765af1-6656-4cbc-945a-ffb56ff8c3dc response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
35262025-02-13T05:08:29.082ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35272025-02-13T05:08:29.082ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
35282025-02-13T05:08:29.111ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35292025-02-13T05:08:29.111ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
35302025-02-13T05:08:29.138ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35312025-02-13T05:08:29.138ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
35322025-02-13T05:08:29.167ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35332025-02-13T05:08:29.167ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
35342025-02-13T05:08:29.180ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56141
35352025-02-13T05:08:29.180ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
35362025-02-13T05:08:29.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35372025-02-13T05:08:29.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35382025-02-13T05:08:29.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
35392025-02-13T05:08:29.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
35402025-02-13T05:08:29.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
35412025-02-13T05:08:29.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
35422025-02-13T05:08:29.180ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 394 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56141 req_id = 5f1995f6-63cd-4f48-96d1-75a172a225b8 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
35432025-02-13T05:08:29.192ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35442025-02-13T05:08:29.192ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
35452025-02-13T05:08:29.216ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
35462025-02-13T05:08:29.217ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
35472025-02-13T05:08:29.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 244070 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36023 req_id = 38569235-c13a-4675-ab19-65fdf93052da response_code = 200 uri = /inventory
35482025-02-13T05:08:35.893ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36516
35492025-02-13T05:08:35.894ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36516 req_id = 8ea056b5-6976-467d-9300-8a650c87f6aa response_code = 200 uri = /vpc-routes
35502025-02-13T05:08:35.976ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
35512025-02-13T05:08:35.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36516 req_id = 3d82c839-e7e1-451d-80ef-8cc064907a98 response_code = 204 uri = /eip-gateways
35522025-02-13T05:08:35.987ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 226 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36516 req_id = 5550fa7b-3ac1-40a7-9b14-9cb27ec38106 response_code = 204 uri = /vpc-routes
35532025-02-13T05:08:54.702ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:63048
35542025-02-13T05:08:54.702ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 33 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:63048 req_id = 5620e307-627c-4d8d-87bf-072d02e4f6e1 response_code = 200 uri = /baseboard
35552025-02-13T05:08:56.467ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47509
35562025-02-13T05:08:56.467ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47509 req_id = 6c8f4dfa-60d0-4013-afbc-4923ffe231cf response_code = 200 uri = /vpc-routes
35572025-02-13T05:08:56.501ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
35582025-02-13T05:08:56.501ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47509 req_id = b6f33f83-3c88-4b6c-96ef-fbeb2473e971 response_code = 204 uri = /eip-gateways
35592025-02-13T05:08:56.520ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 267 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47509 req_id = 358977e8-4f59-40c4-9a1f-0bd7ef8c945c response_code = 204 uri = /vpc-routes
35602025-02-13T05:08:56.579ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63155
35612025-02-13T05:08:56.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63155 req_id = bb4e9914-6352-4d17-9941-e8f653f09c85 response_code = 200 uri = /vpc-routes
35622025-02-13T05:08:56.600ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
35632025-02-13T05:08:56.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63155 req_id = 157d89f4-5142-44d2-a1fc-09a30d1d84e8 response_code = 204 uri = /eip-gateways
35642025-02-13T05:08:56.608ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45125
35652025-02-13T05:08:56.608ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45125 req_id = 3813ba21-f912-4007-a690-dd986847f7f8 response_code = 200 uri = /vpc-routes
35662025-02-13T05:08:56.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63155 req_id = e9a79a26-9c89-4b27-8b64-ddb69bd1e702 response_code = 204 uri = /vpc-routes
35672025-02-13T05:08:56.641ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34721
35682025-02-13T05:08:56.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34721 req_id = ea546cfc-4f5e-4919-9e05-eaa009bd05fd response_code = 200 uri = /v2p
35692025-02-13T05:08:56.669ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
35702025-02-13T05:08:56.669ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45125 req_id = 139865ba-bcdc-4b6a-a07a-094c2b422636 response_code = 204 uri = /eip-gateways
35712025-02-13T05:08:56.678ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 161 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45125 req_id = c0ceb23a-008f-4997-9d6d-6dba200108aa response_code = 204 uri = /vpc-routes
35722025-02-13T05:08:56.706ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37412
35732025-02-13T05:08:56.707ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37412 req_id = 150caf91-bd02-44e7-8ede-d22e570157bb response_code = 200 uri = /v2p
35742025-02-13T05:08:56.736ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42512
35752025-02-13T05:08:56.737ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
35762025-02-13T05:08:56.827ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
35772025-02-13T05:08:56.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 155210 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:42512 req_id = 8a8be85f-69d5-433c-a0a7-9f389ad273f2 response_code = 204 uri = /switch-ports
35782025-02-13T05:08:56.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38089
35792025-02-13T05:08:56.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61541
35802025-02-13T05:08:56.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42861
35812025-02-13T05:08:56.892ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
35822025-02-13T05:08:56.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 4014 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42512 req_id = fdf8d756-ccc0-445d-85ee-55a7cc548101 response_code = 200 uri = /network-bootstore-config
35832025-02-13T05:08:56.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3895 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42861 req_id = 46f0c6f4-3242-4cb1-9e90-b6ac2a27b160 response_code = 200 uri = /v2p
35842025-02-13T05:08:56.936ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
35852025-02-13T05:08:56.963ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
35862025-02-13T05:08:56.963ZINFOSledAgent (bootstore): Writing ledger to /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
35872025-02-13T05:08:56.968ZINFOSledAgent (bootstore): Writing ledger to /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
35882025-02-13T05:08:56.970ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
35892025-02-13T05:08:56.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7813 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42512 req_id = bbaeadd6-103c-437a-9590-1ed54db983aa response_code = 204 uri = /network-bootstore-config
35902025-02-13T05:08:57.002ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
35912025-02-13T05:08:57.002ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 109514 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61541 req_id = 623961e3-fb76-4188-a990-cc4315c3c57c response_code = 204 uri = /switch-ports
35922025-02-13T05:08:57.005ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 646 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61541 req_id = 47180aca-51c2-4b64-84dc-1d763ab5f4f1 response_code = 200 uri = /network-bootstore-config
35932025-02-13T05:08:57.041ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
35942025-02-13T05:08:57.099ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 206660 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38089 req_id = c7dfdc47-e362-4b48-9edd-4d2514f113c0 response_code = 204 uri = /switch-ports
35952025-02-13T05:08:57.099ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38089 req_id = 1df173dd-fb3a-4db7-b60d-22e6a935f1a1 response_code = 200 uri = /network-bootstore-config
35962025-02-13T05:09:24.736ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42844
35972025-02-13T05:09:24.736ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 32 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42844 req_id = dbb5efcd-6ec0-48a8-bc35-3ae3c1256779 response_code = 200 uri = /baseboard
35982025-02-13T05:09:25.173ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:40836 remote_addr = [fd00:1122:3344:101::d]:53516 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
35992025-02-13T05:09:25.228ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 54658 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 1113ad36-8546-48c9-a1fc-862844809cf1 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
36002025-02-13T05:09:26.676ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54331
36012025-02-13T05:09:26.676ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54331 req_id = f17187d5-2ca2-470f-8a07-208ad368905d response_code = 200 uri = /vpc-routes
36022025-02-13T05:09:26.689ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56023
36032025-02-13T05:09:26.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56023 req_id = 70b82fe6-4ee9-4ae1-8c89-4824d27f6529 response_code = 200 uri = /vpc-routes
36042025-02-13T05:09:26.697ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59620
36052025-02-13T05:09:26.697ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59620 req_id = e7e3eb9b-f3b1-4dac-b235-7c0727763758 response_code = 200 uri = /vpc-routes
36062025-02-13T05:09:26.742ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36072025-02-13T05:09:26.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56023 req_id = 1b21ebde-1b98-4574-951a-47efcf4188ac response_code = 204 uri = /eip-gateways
36082025-02-13T05:09:26.744ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36092025-02-13T05:09:26.744ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 29 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54331 req_id = 321a531a-dabc-42ef-8a1a-e128c610523c response_code = 204 uri = /eip-gateways
36102025-02-13T05:09:26.753ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36112025-02-13T05:09:26.753ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:59620 req_id = 1e659094-ddb1-4d19-8ed7-2f5021c6664b response_code = 204 uri = /eip-gateways
36122025-02-13T05:09:26.754ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 237 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56023 req_id = 0b20ef35-9371-4f2b-abae-62cb3a8e5d0a response_code = 204 uri = /vpc-routes
36132025-02-13T05:09:26.757ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54331 req_id = 53cd1c98-c46e-44ce-9d78-b1d0b08b9f2c response_code = 204 uri = /vpc-routes
36142025-02-13T05:09:26.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 210 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:59620 req_id = 95273d94-3d71-40ac-8dba-13982049ecd6 response_code = 204 uri = /vpc-routes
36152025-02-13T05:09:26.774ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46285
36162025-02-13T05:09:26.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46285 req_id = f03184ac-6bb2-41bb-b77d-6563dce6e87c response_code = 200 uri = /v2p
36172025-02-13T05:09:26.824ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55408
36182025-02-13T05:09:26.824ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55408 req_id = 2d456aae-2af1-4652-8158-a79b791074cd response_code = 200 uri = /v2p
36192025-02-13T05:09:26.865ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62387
36202025-02-13T05:09:26.865ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
36212025-02-13T05:09:26.906ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
36222025-02-13T05:09:26.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 95758 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62387 req_id = 4bb154e6-80f2-438f-bf2c-6c903dcb1194 response_code = 204 uri = /switch-ports
36232025-02-13T05:09:26.961ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41292
36242025-02-13T05:09:26.961ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40492
36252025-02-13T05:09:26.961ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35273
36262025-02-13T05:09:26.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62387 req_id = a252a1d5-7c97-4107-acfe-0e45224cc164 response_code = 200 uri = /network-bootstore-config
36272025-02-13T05:09:26.961ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
36282025-02-13T05:09:26.965ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3847 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35273 req_id = a8ccd144-629c-49e5-a74c-fa32fd363d8b response_code = 200 uri = /v2p
36292025-02-13T05:09:26.998ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
36302025-02-13T05:09:27.050ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
36312025-02-13T05:09:27.050ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 88708 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41292 req_id = 0a10531b-b0a3-4e02-9ec1-460e53661853 response_code = 204 uri = /switch-ports
36322025-02-13T05:09:27.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 519 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41292 req_id = 1881e810-a071-451c-8135-d367b37b742c response_code = 200 uri = /network-bootstore-config
36332025-02-13T05:09:27.082ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
36342025-02-13T05:09:27.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 171192 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40492 req_id = 0d310400-1603-4542-9d75-433339f09866 response_code = 204 uri = /switch-ports
36352025-02-13T05:09:27.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40492 req_id = 31b6f253-13bd-406d-929e-0b2f6d2752ba response_code = 200 uri = /network-bootstore-config
36362025-02-13T05:09:54.770ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36385
36372025-02-13T05:09:54.770ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 35 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36385 req_id = f593b660-9938-450b-9231-40cd185ea948 response_code = 200 uri = /baseboard
36382025-02-13T05:09:55.178ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7219 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 3c87bfee-a0fd-4691-a5c6-861ec7d70cb2 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
36392025-02-13T05:09:56.475ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48309
36402025-02-13T05:09:56.475ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48309 req_id = df3e2be3-8c7d-4a1c-a5b0-eac0d60c374a response_code = 200 uri = /vpc-routes
36412025-02-13T05:09:56.496ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36422025-02-13T05:09:56.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48309 req_id = f1b29701-fdd8-4798-9f36-9ab43388dd26 response_code = 204 uri = /eip-gateways
36432025-02-13T05:09:56.506ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 218 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48309 req_id = 1e87a8d2-43d6-45a8-9850-bfcd5cd57fde response_code = 204 uri = /vpc-routes
36442025-02-13T05:09:56.525ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53843
36452025-02-13T05:09:56.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53843 req_id = ff26676b-4cb2-4363-a526-66e51386c7b7 response_code = 200 uri = /vpc-routes
36462025-02-13T05:09:56.567ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49752
36472025-02-13T05:09:56.567ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49752 req_id = e636dfd8-ef11-4c4f-9365-c489cf764bb4 response_code = 200 uri = /v2p
36482025-02-13T05:09:56.585ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36492025-02-13T05:09:56.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53843 req_id = cb0c01ad-2e3c-489c-87d0-b7324904933a response_code = 204 uri = /eip-gateways
36502025-02-13T05:09:56.596ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58142
36512025-02-13T05:09:56.597ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58142 req_id = 799efe03-d081-4810-ba1d-ae59e0757124 response_code = 200 uri = /v2p
36522025-02-13T05:09:56.603ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 251 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53843 req_id = 54c9a676-4cad-46a6-bf3c-15e63bd2fb62 response_code = 204 uri = /vpc-routes
36532025-02-13T05:09:56.620ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60169
36542025-02-13T05:09:56.620ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
36552025-02-13T05:09:56.659ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
36562025-02-13T05:09:56.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 96121 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60169 req_id = 0c8b3188-6207-4f5c-82ff-207114ccc537 response_code = 204 uri = /switch-ports
36572025-02-13T05:09:56.716ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35625
36582025-02-13T05:09:56.716ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61731
36592025-02-13T05:09:56.716ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45434
36602025-02-13T05:09:56.716ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50231
36612025-02-13T05:09:56.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 288 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60169 req_id = 518392cf-1d2f-4429-9e22-853c3cf854d3 response_code = 200 uri = /network-bootstore-config
36622025-02-13T05:09:56.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 251 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45434 req_id = 5eeb8d45-7163-4b46-978d-555580cc62a6 response_code = 200 uri = /v2p
36632025-02-13T05:09:56.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 215 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35625 req_id = f11ac2e2-4e8b-4456-80b8-ace013193996 response_code = 200 uri = /vpc-routes
36642025-02-13T05:09:56.717ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
36652025-02-13T05:09:56.758ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
36662025-02-13T05:09:56.761ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36672025-02-13T05:09:56.761ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 547 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35625 req_id = ac10d6ac-e232-4a93-9faa-33792aee72bd response_code = 204 uri = /eip-gateways
36682025-02-13T05:09:56.773ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 786 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35625 req_id = 883afd02-199c-4341-a65f-1ac8635659ff response_code = 204 uri = /vpc-routes
36692025-02-13T05:09:56.814ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
36702025-02-13T05:09:56.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 96942 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61731 req_id = b7a1c492-f893-4001-8fc9-e04b14e54495 response_code = 204 uri = /switch-ports
36712025-02-13T05:09:56.817ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 716 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61731 req_id = d363282f-9bc8-432f-aa23-1ceeb9b52294 response_code = 200 uri = /network-bootstore-config
36722025-02-13T05:09:56.850ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
36732025-02-13T05:09:56.905ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 188441 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50231 req_id = 1b8080f2-5457-4a0c-9647-c28a4bb220d9 response_code = 204 uri = /switch-ports
36742025-02-13T05:09:56.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50231 req_id = 3b976aef-685f-478f-99f7-f22c99fd6996 response_code = 200 uri = /network-bootstore-config
36752025-02-13T05:10:24.803ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44864
36762025-02-13T05:10:24.803ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 51 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44864 req_id = ac9801dc-276c-412f-a5b5-de9ebe21959d response_code = 200 uri = /baseboard
36772025-02-13T05:10:25.179ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7604 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = b250a0c0-b4fc-44a3-a20f-a439c80f4a8a response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
36782025-02-13T05:10:26.461ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44864
36792025-02-13T05:10:26.461ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44864 req_id = a757b423-91ca-4fcb-a942-19741d174423 response_code = 200 uri = /vpc-routes
36802025-02-13T05:10:26.621ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36812025-02-13T05:10:26.621ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44864 req_id = dbc04e35-e83b-404f-be97-2709700eb7a2 response_code = 204 uri = /eip-gateways
36822025-02-13T05:10:26.632ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 235 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44864 req_id = 3ffe08e6-2594-4a01-a29e-c8a930694934 response_code = 204 uri = /vpc-routes
36832025-02-13T05:10:26.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36935
36842025-02-13T05:10:26.690ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36935 req_id = f00cba87-d8f1-4382-a546-3c33586ce9d7 response_code = 200 uri = /vpc-routes
36852025-02-13T05:10:26.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44732
36862025-02-13T05:10:26.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44732 req_id = a888d453-2cdd-4e05-97cf-d025fd7d59a4 response_code = 200 uri = /v2p
36872025-02-13T05:10:26.741ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
36882025-02-13T05:10:26.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36935 req_id = a49f1b9e-e8cb-4178-872a-e447bb929561 response_code = 204 uri = /eip-gateways
36892025-02-13T05:10:26.753ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36935 req_id = b3c0b2ae-77f2-4e0f-af02-3c3c3d50cd48 response_code = 204 uri = /vpc-routes
36902025-02-13T05:10:26.764ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44552
36912025-02-13T05:10:26.765ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44552 req_id = 2c0f34be-3140-4af3-937b-6e292c336f8a response_code = 200 uri = /v2p
36922025-02-13T05:10:26.786ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43043
36932025-02-13T05:10:26.786ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43043 req_id = 28829ce9-f3d5-4ee7-b429-f03c9a1a3bf6 response_code = 200 uri = /v2p
36942025-02-13T05:10:26.825ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40243
36952025-02-13T05:10:26.826ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
36962025-02-13T05:10:26.867ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
36972025-02-13T05:10:26.927ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101790 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40243 req_id = 2b161724-e13b-48e7-a476-340b05a37a28 response_code = 204 uri = /switch-ports
36982025-02-13T05:10:26.927ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58863
36992025-02-13T05:10:26.927ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51759
37002025-02-13T05:10:26.927ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36064
37012025-02-13T05:10:26.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58863 req_id = 618fc2b2-fdd9-4141-98c2-0cfb72f66dc1 response_code = 200 uri = /vpc-routes
37022025-02-13T05:10:26.928ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37032025-02-13T05:10:26.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 224 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40243 req_id = bd07d32a-eede-4238-ae33-2b6f27e9f198 response_code = 200 uri = /network-bootstore-config
37042025-02-13T05:10:26.969ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37052025-02-13T05:10:26.972ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
37062025-02-13T05:10:26.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 575 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58863 req_id = f7fe1037-a048-49c1-a953-31415c3f35b3 response_code = 204 uri = /eip-gateways
37072025-02-13T05:10:26.985ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 879 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58863 req_id = cca60fec-7078-4eb2-abf1-5d0e005ffbe7 response_code = 204 uri = /vpc-routes
37082025-02-13T05:10:27.031ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37092025-02-13T05:10:27.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 103370 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:36064 req_id = c4d8316e-59c0-4ceb-b274-68f5bce3bc35 response_code = 204 uri = /switch-ports
37102025-02-13T05:10:27.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 595 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36064 req_id = 68f738e8-8bb6-43f1-9d22-cc34e07a66d9 response_code = 200 uri = /network-bootstore-config
37112025-02-13T05:10:27.065ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37122025-02-13T05:10:27.123ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 190153 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51759 req_id = a1d9e5d5-b650-4dd0-bcf3-a24d06346274 response_code = 204 uri = /switch-ports
37132025-02-13T05:10:27.123ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51759 req_id = 97529f74-488d-4e72-856a-5d21a2acb8eb response_code = 200 uri = /network-bootstore-config
37142025-02-13T05:10:48.017ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
37152025-02-13T05:10:54.841ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64415
37162025-02-13T05:10:54.841ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 31 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64415 req_id = 3719e6c0-a9bc-4fc8-bee0-be8ec6d762db response_code = 200 uri = /baseboard
37172025-02-13T05:10:55.178ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7816 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 14affbea-d138-4071-8a2f-5ec2429dcfa9 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
37182025-02-13T05:10:56.492ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43834
37192025-02-13T05:10:56.492ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43834 req_id = 44c9b3dc-cdc8-44ab-ae68-6b156fd11e6e response_code = 200 uri = /vpc-routes
37202025-02-13T05:10:56.520ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50278
37212025-02-13T05:10:56.520ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50278 req_id = 6faf100c-aa79-4c94-ae05-0620faa82096 response_code = 200 uri = /vpc-routes
37222025-02-13T05:10:56.543ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
37232025-02-13T05:10:56.543ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50278 req_id = 6e4700ab-74cb-4101-91c7-0f7b07f2da87 response_code = 204 uri = /eip-gateways
37242025-02-13T05:10:56.552ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
37252025-02-13T05:10:56.552ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43834 req_id = 747df3ee-0bad-4d29-9f23-5dbcf27b3ee4 response_code = 204 uri = /eip-gateways
37262025-02-13T05:10:56.565ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43834 req_id = 9c3d9fa9-ee2b-482d-8b21-34ad56363049 response_code = 204 uri = /vpc-routes
37272025-02-13T05:10:56.606ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59420
37282025-02-13T05:10:56.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59420 req_id = 0535130d-d830-41e0-af12-a94520d9037f response_code = 200 uri = /v2p
37292025-02-13T05:10:56.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 286 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50278 req_id = 58c69ea3-a8c4-486b-b978-bd3dd734d5ce response_code = 204 uri = /vpc-routes
37302025-02-13T05:10:56.638ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36595
37312025-02-13T05:10:56.638ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37322025-02-13T05:10:56.679ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37332025-02-13T05:10:56.738ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 99884 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36595 req_id = 82a0ff16-0f50-4008-a779-0e3141eb45c1 response_code = 204 uri = /switch-ports
37342025-02-13T05:10:56.738ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35983
37352025-02-13T05:10:56.739ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54479
37362025-02-13T05:10:56.739ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35484
37372025-02-13T05:10:56.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 250 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36595 req_id = 1f91f1de-8744-42b5-bc34-28ca061258f0 response_code = 200 uri = /network-bootstore-config
37382025-02-13T05:10:56.739ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37392025-02-13T05:10:56.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35484 req_id = 4a645889-d7a4-4e65-b8df-fb9dbda166a4 response_code = 200 uri = /vpc-routes
37402025-02-13T05:10:56.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35983 req_id = 79bd14f6-c93e-474d-9476-ad754e5cf676 response_code = 200 uri = /v2p
37412025-02-13T05:10:56.755ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
37422025-02-13T05:10:56.755ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35484 req_id = 214572ff-8f5e-4a47-9429-a5a44efcb944 response_code = 204 uri = /eip-gateways
37432025-02-13T05:10:56.774ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37442025-02-13T05:10:56.823ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37223
37452025-02-13T05:10:56.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37223 req_id = 046beb0b-da59-42b2-8050-37163afb20c3 response_code = 200 uri = /v2p
37462025-02-13T05:10:56.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 87016 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54479 req_id = f22e2575-2c90-4ed7-ae27-ebf589ad715b response_code = 204 uri = /switch-ports
37472025-02-13T05:10:56.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54479 req_id = c630462e-cae4-4919-9bdb-692ff5a0a244 response_code = 200 uri = /network-bootstore-config
37482025-02-13T05:10:56.827ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 258 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35484 req_id = c443d113-e87b-448a-af98-6914295fe740 response_code = 204 uri = /vpc-routes
37492025-02-13T05:10:56.838ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59757
37502025-02-13T05:10:56.838ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37512025-02-13T05:10:56.871ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37522025-02-13T05:10:56.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 83395 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59757 req_id = d18cbea5-f829-4d47-a986-0f9a2a41860b response_code = 204 uri = /switch-ports
37532025-02-13T05:10:56.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59757 req_id = d55e0d3c-6f6c-4742-8e34-6fd66ed9d08e response_code = 200 uri = /network-bootstore-config
37542025-02-13T05:11:24.874ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40370
37552025-02-13T05:11:24.874ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 38 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40370 req_id = 6eb33c21-83fc-4520-8b94-d9fcb64e78d2 response_code = 200 uri = /baseboard
37562025-02-13T05:11:25.178ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7662 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = b4997f92-048e-4fa3-b877-a47155cb0c2e response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
37572025-02-13T05:11:26.535ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60260
37582025-02-13T05:11:26.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 409 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60260 req_id = 3e042a78-1636-4d17-8ec8-95ccf049576f response_code = 200 uri = /vpc-routes
37592025-02-13T05:11:26.629ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
37602025-02-13T05:11:26.629ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60260 req_id = b27eb040-fb35-4cbc-b074-d5a5c2f89a2e response_code = 204 uri = /eip-gateways
37612025-02-13T05:11:26.643ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 323 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60260 req_id = d19dce02-d46e-4181-bb06-cd7d6b0bbba4 response_code = 204 uri = /vpc-routes
37622025-02-13T05:11:26.680ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60135
37632025-02-13T05:11:26.680ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 156 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60135 req_id = 8cbf8e53-c7d0-421c-bb7f-d9d9270ccc24 response_code = 200 uri = /v2p
37642025-02-13T05:11:26.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36915
37652025-02-13T05:11:26.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36915 req_id = 4973e3b7-fa09-42fd-8930-e0ff192cb3f3 response_code = 200 uri = /vpc-routes
37662025-02-13T05:11:26.708ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
37672025-02-13T05:11:26.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36915 req_id = d5429aed-af11-4f83-a5fa-82e7c96e4d8c response_code = 204 uri = /eip-gateways
37682025-02-13T05:11:26.720ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 567 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36915 req_id = f6d00e12-d6c5-48e2-ad4e-a798f7392c5e response_code = 204 uri = /vpc-routes
37692025-02-13T05:11:26.761ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39580
37702025-02-13T05:11:26.761ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37712025-02-13T05:11:26.814ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37722025-02-13T05:11:26.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 121801 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39580 req_id = 37180927-b80d-4d26-a203-f8ce0b0d27f8 response_code = 204 uri = /switch-ports
37732025-02-13T05:11:26.883ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41061
37742025-02-13T05:11:26.883ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57685
37752025-02-13T05:11:26.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 293 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57685 req_id = 3b3dc630-de47-4def-bba3-057895683603 response_code = 200 uri = /v2p
37762025-02-13T05:11:26.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41061 req_id = b01f282e-78e4-46fb-9d84-a9c23dbcec2e response_code = 200 uri = /vpc-routes
37772025-02-13T05:11:26.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 276 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39580 req_id = e4c7b3dc-0a14-4ca9-b9d9-2613407e9f1e response_code = 200 uri = /network-bootstore-config
37782025-02-13T05:11:26.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64612
37792025-02-13T05:11:26.892ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37802025-02-13T05:11:26.945ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37812025-02-13T05:11:27.033ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 141044 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64612 req_id = 7ca9b41d-e396-4f64-b3d0-41fdca1d149d response_code = 204 uri = /switch-ports
37822025-02-13T05:11:27.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51127
37832025-02-13T05:11:27.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 247 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51127 req_id = 595203ea-70db-45c7-abc5-5a8b55119f4e response_code = 200 uri = /v2p
37842025-02-13T05:11:27.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 276 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64612 req_id = d88e31b9-9c36-433a-9099-87ab30ecce8d response_code = 200 uri = /network-bootstore-config
37852025-02-13T05:11:27.034ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
37862025-02-13T05:11:27.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41061 req_id = 54c57f16-c6c1-4081-a64e-7001f832c7b5 response_code = 204 uri = /eip-gateways
37872025-02-13T05:11:27.041ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34414
37882025-02-13T05:11:27.041ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
37892025-02-13T05:11:27.095ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
37902025-02-13T05:11:27.176ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 135121 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34414 req_id = 6467485a-187e-4bda-8fc6-8c5a0fcd1f1b response_code = 204 uri = /switch-ports
37912025-02-13T05:11:27.177ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 230 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34414 req_id = 8740f3fa-26d8-4bb9-84f2-b2bb58695f93 response_code = 200 uri = /network-bootstore-config
37922025-02-13T05:11:27.177ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 519 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41061 req_id = e3b77804-d8b9-4329-8c2c-d59831ea17a9 response_code = 204 uri = /vpc-routes
37932025-02-13T05:11:54.909ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44579
37942025-02-13T05:11:54.909ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 36 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44579 req_id = 84046627-ad28-435d-a6ca-e1398b557b8e response_code = 200 uri = /baseboard
37952025-02-13T05:11:55.176ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 6599 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 1c0bdb08-c4e0-4de6-9b95-656ded5c5579 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
37962025-02-13T05:11:56.462ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61542
37972025-02-13T05:11:56.462ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61542 req_id = e32b1eb3-1755-4a04-819f-3aeaeffdcb8c response_code = 200 uri = /vpc-routes
37982025-02-13T05:11:56.471ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65091
37992025-02-13T05:11:56.471ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65091 req_id = b9b66293-b03e-4c82-ae15-1373e28e7049 response_code = 200 uri = /vpc-routes
38002025-02-13T05:11:56.539ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38012025-02-13T05:11:56.540ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61542 req_id = 051a8d6e-ac0f-4b4f-abef-81182fa78abd response_code = 204 uri = /eip-gateways
38022025-02-13T05:11:56.541ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38032025-02-13T05:11:56.541ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65091 req_id = 62fc2b6e-4209-4e1c-b13c-290167923580 response_code = 204 uri = /eip-gateways
38042025-02-13T05:11:56.557ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 380 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61542 req_id = d34abc34-81da-4046-a4be-67c62f3efe84 response_code = 204 uri = /vpc-routes
38052025-02-13T05:11:56.562ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65091 req_id = 066d4531-0f27-4720-8784-8d76b78107d0 response_code = 204 uri = /vpc-routes
38062025-02-13T05:11:56.568ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56353
38072025-02-13T05:11:56.568ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38082025-02-13T05:11:56.608ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
38092025-02-13T05:11:56.668ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 100222 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56353 req_id = e1975aa3-ef22-40fe-bd49-927ff1608cc0 response_code = 204 uri = /switch-ports
38102025-02-13T05:11:56.668ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49034
38112025-02-13T05:11:56.668ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42359
38122025-02-13T05:11:56.669ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38132025-02-13T05:11:56.672ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3833 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49034 req_id = 9d8bc9e8-e112-4549-9e33-509085cf301a response_code = 200 uri = /v2p
38142025-02-13T05:11:56.673ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3989 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56353 req_id = 3f7c543a-8e4d-442d-a387-44544c1de83b response_code = 200 uri = /network-bootstore-config
38152025-02-13T05:11:56.710ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
38162025-02-13T05:11:56.752ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39038
38172025-02-13T05:11:56.752ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39038 req_id = 3f3dd0eb-5208-49da-a813-6ab74f4fcc5c response_code = 200 uri = /vpc-routes
38182025-02-13T05:11:56.760ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63092
38192025-02-13T05:11:56.760ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 267 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63092 req_id = 0c492e23-ede7-461b-8770-a12a05441594 response_code = 200 uri = /v2p
38202025-02-13T05:11:56.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 98615 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:42359 req_id = e662b650-636e-449d-9f4f-7d3cc2ec187d response_code = 204 uri = /switch-ports
38212025-02-13T05:11:56.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42359 req_id = 0f1ebc43-c723-49b1-bd06-2b494caf5c0d response_code = 200 uri = /network-bootstore-config
38222025-02-13T05:11:56.768ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38232025-02-13T05:11:56.768ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:39038 req_id = a6017ac3-4f15-4b98-9e16-b85876f68a0f response_code = 204 uri = /eip-gateways
38242025-02-13T05:11:56.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:39038 req_id = b20ba312-da04-4211-b338-b97c555193f3 response_code = 204 uri = /vpc-routes
38252025-02-13T05:11:56.814ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36525
38262025-02-13T05:11:56.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36525 req_id = a9f5e73c-0bcb-44b4-b765-fc0da483bd3d response_code = 200 uri = /v2p
38272025-02-13T05:11:56.855ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55544
38282025-02-13T05:11:56.855ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38292025-02-13T05:11:56.894ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
38302025-02-13T05:11:56.957ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 102400 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:55544 req_id = caee215f-a9fe-4573-9d2d-21bb09d8d5f7 response_code = 204 uri = /switch-ports
38312025-02-13T05:11:56.958ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55544 req_id = 929d1717-51bf-4a5b-a906-3ffe4114e300 response_code = 200 uri = /network-bootstore-config
38322025-02-13T05:12:24.943ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:57666
38332025-02-13T05:12:24.943ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 78 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:57666 req_id = 1d633e65-acac-42d1-be4e-0a615844acb5 response_code = 200 uri = /baseboard
38342025-02-13T05:12:25.177ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 6930 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = cdbcea8d-6f3d-49df-b3fb-13202a48a759 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
38352025-02-13T05:12:26.616ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54645
38362025-02-13T05:12:26.616ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54645 req_id = 484b4bb8-85d0-462a-9c36-63a0fbb22ac0 response_code = 200 uri = /vpc-routes
38372025-02-13T05:12:26.619ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65370
38382025-02-13T05:12:26.619ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65370 req_id = c8ac84af-4dff-4b67-950f-c23fdb07bdea response_code = 200 uri = /vpc-routes
38392025-02-13T05:12:26.668ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38402025-02-13T05:12:26.668ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54645 req_id = 71f86403-0489-4eea-8b37-8aa3803b0966 response_code = 204 uri = /eip-gateways
38412025-02-13T05:12:26.686ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38422025-02-13T05:12:26.686ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65370 req_id = e9813dd3-4e72-4a71-90b0-9b87569943bf response_code = 204 uri = /eip-gateways
38432025-02-13T05:12:26.686ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 369 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54645 req_id = 4823c043-8608-4b3c-8a11-fbacdfe873f3 response_code = 204 uri = /vpc-routes
38442025-02-13T05:12:26.702ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 362 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65370 req_id = 10f70ae9-2742-44e1-9224-d13ea0b0523f response_code = 204 uri = /vpc-routes
38452025-02-13T05:12:26.737ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46743
38462025-02-13T05:12:26.737ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46743 req_id = b6e88349-aa53-4317-949d-25af13d52542 response_code = 200 uri = /vpc-routes
38472025-02-13T05:12:26.785ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38482025-02-13T05:12:26.785ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46743 req_id = 9c850953-a24e-4d8b-8349-8de9201f894c response_code = 204 uri = /eip-gateways
38492025-02-13T05:12:26.792ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46743 req_id = 79cbe103-2c87-4fb9-8bfb-f4687f4189d4 response_code = 204 uri = /vpc-routes
38502025-02-13T05:12:26.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38690
38512025-02-13T05:12:26.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38690 req_id = dde842b8-16a8-45ac-97d6-b8be4d8a1dfc response_code = 200 uri = /v2p
38522025-02-13T05:12:26.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57785
38532025-02-13T05:12:26.795ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38542025-02-13T05:12:26.836ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
38552025-02-13T05:12:26.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 100777 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57785 req_id = 13bbba10-ad77-4681-891b-748fe5facb8c response_code = 204 uri = /switch-ports
38562025-02-13T05:12:26.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52602
38572025-02-13T05:12:26.897ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50287
38582025-02-13T05:12:26.897ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51154
38592025-02-13T05:12:26.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57785 req_id = 74f7f0a2-80bd-40a3-90fc-da868dc59e1b response_code = 200 uri = /network-bootstore-config
38602025-02-13T05:12:26.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 237 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52602 req_id = e54f187d-5f10-4bd0-b093-93d6b38e8fef response_code = 200 uri = /v2p
38612025-02-13T05:12:26.897ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38622025-02-13T05:12:26.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50287 req_id = 6f0a1e75-f325-4925-990b-4716fa6c46cb response_code = 200 uri = /v2p
38632025-02-13T05:12:26.912ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35699
38642025-02-13T05:12:26.937ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
38652025-02-13T05:12:26.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 98859 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51154 req_id = d2a7216c-3865-4b97-a302-c446f30619e6 response_code = 204 uri = /switch-ports
38662025-02-13T05:12:26.996ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38672025-02-13T05:12:27.000ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 611 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51154 req_id = 67814eee-1078-4cc1-a91e-6a08073702db response_code = 200 uri = /network-bootstore-config
38682025-02-13T05:12:27.033ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
38692025-02-13T05:12:27.085ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 172454 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35699 req_id = 3828c7f5-9f26-4d17-b81b-ad9657de08e8 response_code = 204 uri = /switch-ports
38702025-02-13T05:12:27.085ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35699 req_id = 088a7cfe-e283-4494-b8b8-0cb63d1d1b05 response_code = 200 uri = /network-bootstore-config
38712025-02-13T05:12:54.977ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35395
38722025-02-13T05:12:54.977ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 41 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35395 req_id = f396c966-03e1-4a35-a41a-38c8160de132 response_code = 200 uri = /baseboard
38732025-02-13T05:12:55.177ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7447 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 1cc51d08-5d02-4423-bfa3-62119dfd03a0 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
38742025-02-13T05:12:56.515ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36943
38752025-02-13T05:12:56.515ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36943 req_id = b4da9d7b-f500-4ff0-a8b3-ebbe8a233e24 response_code = 200 uri = /vpc-routes
38762025-02-13T05:12:56.527ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64090
38772025-02-13T05:12:56.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64090 req_id = 04b2228a-aba4-4cae-a885-972fc1f7de2a response_code = 200 uri = /vpc-routes
38782025-02-13T05:12:56.555ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38792025-02-13T05:12:56.555ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 191 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64090 req_id = bb2e786e-83c8-4564-a78e-d239e85554a3 response_code = 204 uri = /eip-gateways
38802025-02-13T05:12:56.556ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64015
38812025-02-13T05:12:56.556ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64015 req_id = 39a4b470-1aa2-4bba-bf63-670d07fcaff1 response_code = 200 uri = /v2p
38822025-02-13T05:12:56.566ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64090 req_id = c3951cf2-e3a1-47cf-8e1f-2d0b513c4b65 response_code = 204 uri = /vpc-routes
38832025-02-13T05:12:56.576ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38842025-02-13T05:12:56.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36943 req_id = 4111256e-5ac9-4dbc-b828-8871856a6e18 response_code = 204 uri = /eip-gateways
38852025-02-13T05:12:56.580ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49115
38862025-02-13T05:12:56.581ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38872025-02-13T05:12:56.626ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
38882025-02-13T05:12:56.688ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 107390 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49115 req_id = 1fac142e-bc35-44a9-827c-12612833ef97 response_code = 204 uri = /switch-ports
38892025-02-13T05:12:56.688ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54852
38902025-02-13T05:12:56.688ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49115 req_id = b6a01981-2956-468e-b918-5e3d8594435e response_code = 200 uri = /network-bootstore-config
38912025-02-13T05:12:56.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 541 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36943 req_id = 227a9fd8-63c3-4b52-8d02-a1966f5b0bec response_code = 204 uri = /vpc-routes
38922025-02-13T05:12:56.689ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
38932025-02-13T05:12:56.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55567
38942025-02-13T05:12:56.694ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55567 req_id = 8768fff2-d5b8-449d-a6e8-06606d03d9f1 response_code = 200 uri = /vpc-routes
38952025-02-13T05:12:56.710ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
38962025-02-13T05:12:56.710ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55567 req_id = f6fff565-ebf3-4ff3-92d1-380d90b0bafb response_code = 204 uri = /eip-gateways
38972025-02-13T05:12:56.720ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 594 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55567 req_id = 5247b450-1aef-42e1-bf1d-11b313fbc6a2 response_code = 204 uri = /vpc-routes
38982025-02-13T05:12:56.721ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53263
38992025-02-13T05:12:56.721ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53263 req_id = a82809c3-9d73-40c3-827e-13e953c01db9 response_code = 200 uri = /v2p
39002025-02-13T05:12:56.727ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
39012025-02-13T05:12:56.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61565
39022025-02-13T05:12:56.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61565 req_id = e3f1c2c5-593f-4a1d-8231-5b882ed873a7 response_code = 200 uri = /v2p
39032025-02-13T05:12:56.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 90115 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54852 req_id = 46b6039e-ec41-4292-8170-29909cad4a78 response_code = 204 uri = /switch-ports
39042025-02-13T05:12:56.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54852 req_id = 604e3bd4-877d-419d-ab87-c5e4a9b3988d response_code = 200 uri = /network-bootstore-config
39052025-02-13T05:12:56.838ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48452
39062025-02-13T05:12:56.839ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
39072025-02-13T05:12:56.871ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
39082025-02-13T05:12:56.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 81828 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48452 req_id = e9fdadd3-a9a5-4fdf-b30d-7ec6f5be8662 response_code = 204 uri = /switch-ports
39092025-02-13T05:12:56.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48452 req_id = 6e92251a-f05d-4495-9e59-59917c009dd0 response_code = 200 uri = /network-bootstore-config
39102025-02-13T05:13:25.009ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46319
39112025-02-13T05:13:25.009ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 38 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46319 req_id = 098976da-b32f-4b92-bead-47d9c1e95f58 response_code = 200 uri = /baseboard
39122025-02-13T05:13:25.179ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 9901 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = fe21a4b0-25ef-4ffa-b756-91e06df5cb0f response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
39132025-02-13T05:13:26.682ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48194
39142025-02-13T05:13:26.682ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48194 req_id = 7a73c706-29cd-4fd1-9303-3d82b59a5908 response_code = 200 uri = /vpc-routes
39152025-02-13T05:13:26.688ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47758
39162025-02-13T05:13:26.688ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
39172025-02-13T05:13:26.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 438 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47758 req_id = c0f32cb2-5cdd-45b1-a49e-4e1e4ce18d0b response_code = 200 uri = /artifacts
39182025-02-13T05:13:26.724ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36351
39192025-02-13T05:13:26.724ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36351 req_id = 7ca37b12-e180-4355-beea-9384c348d945 response_code = 200 uri = /vpc-routes
39202025-02-13T05:13:26.752ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54724
39212025-02-13T05:13:26.752ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
39222025-02-13T05:13:26.753ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 401 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54724 req_id = 761e4b42-b0df-4f44-9389-671f21e3734d response_code = 200 uri = /artifacts
39232025-02-13T05:13:26.769ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
39242025-02-13T05:13:26.769ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48194 req_id = 5e8bf733-4ee9-418c-8a02-0da644b3e36a response_code = 204 uri = /eip-gateways
39252025-02-13T05:13:26.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 299 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48194 req_id = 7dfa7894-e2ae-4365-a4f5-a69a01398bd7 response_code = 204 uri = /vpc-routes
39262025-02-13T05:13:26.781ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
39272025-02-13T05:13:26.781ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36351 req_id = 88c45758-a078-48eb-b331-4e6063b0277f response_code = 204 uri = /eip-gateways
39282025-02-13T05:13:26.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36527
39292025-02-13T05:13:26.795ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
39302025-02-13T05:13:26.842ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
39312025-02-13T05:13:26.909ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 114642 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36527 req_id = d9bffdea-714d-4aee-9352-c5655dc292c1 response_code = 204 uri = /switch-ports
39322025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62424
39332025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49063
39342025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59944
39352025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47758
39362025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 358 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36527 req_id = c65f0810-9431-4f12-8028-1bf25d9c662f response_code = 200 uri = /network-bootstore-config
39372025-02-13T05:13:26.910ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
39382025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 266 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59944 req_id = 4b4eed84-f609-43a4-9201-cd50cb9c2c88 response_code = 200 uri = /v2p
39392025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 395 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49063 req_id = 913f564e-e857-42ea-895d-ceb6ea0947ee response_code = 200 uri = /v2p
39402025-02-13T05:13:26.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 680 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36351 req_id = 537a3260-c443-455e-8b0d-e6228f9b7e76 response_code = 204 uri = /vpc-routes
39412025-02-13T05:13:26.911ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 493 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47758 req_id = b0542dff-9ae7-4374-9b58-3db35b76607a response_code = 200 uri = /artifacts
39422025-02-13T05:13:26.911ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 550 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62424 req_id = 9885d989-b555-444c-ba16-eb624b73d88a response_code = 200 uri = /vpc-routes
39432025-02-13T05:13:26.963ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
39442025-02-13T05:13:26.963ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 778 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62424 req_id = 0e269d11-7c82-4540-94cc-abc59403e983 response_code = 204 uri = /eip-gateways
39452025-02-13T05:13:26.971ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46813
39462025-02-13T05:13:26.972ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
39472025-02-13T05:13:27.016ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
39482025-02-13T05:13:27.079ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 107293 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46813 req_id = ddb0c749-dfc6-4123-b9fc-5949456275ef response_code = 204 uri = /switch-ports
39492025-02-13T05:13:27.079ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50225
39502025-02-13T05:13:27.079ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 325 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62424 req_id = 38adf1aa-09f9-4d55-984b-c128344b2fc1 response_code = 204 uri = /vpc-routes
39512025-02-13T05:13:27.080ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46813 req_id = 4c8bb603-00e3-4c79-a982-41334598717f response_code = 200 uri = /network-bootstore-config
39522025-02-13T05:13:27.080ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 232 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50225 req_id = f7771d5c-c4c6-4042-8e89-5b95add33617 response_code = 200 uri = /v2p
39532025-02-13T05:13:27.084ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63353
39542025-02-13T05:13:27.085ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
39552025-02-13T05:13:27.085ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39562025-02-13T05:13:27.085ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39572025-02-13T05:13:27.085ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39582025-02-13T05:13:27.085ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
39592025-02-13T05:13:27.085ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
39602025-02-13T05:13:27.085ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
39612025-02-13T05:13:27.085ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 474 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63353 req_id = efda7b21-f397-42dc-9e09-b73ca5b3b31d response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
39622025-02-13T05:13:27.160ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37780
39632025-02-13T05:13:27.161ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
39642025-02-13T05:13:27.202ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
39652025-02-13T05:13:27.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 99760 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37780 req_id = 4889ebfd-291c-448a-888e-8a31f44b90a5 response_code = 204 uri = /switch-ports
39662025-02-13T05:13:27.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60072
39672025-02-13T05:13:27.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54252
39682025-02-13T05:13:27.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 324 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37780 req_id = ef1b7c7b-894f-48f5-ae89-1489b2b1bd0b response_code = 200 uri = /network-bootstore-config
39692025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
39702025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
39712025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39722025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39732025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39742025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
39752025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
39762025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
39772025-02-13T05:13:27.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 493 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54252 req_id = 6683efb7-2344-4d76-ac06-8c4c172565e6 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
39782025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39792025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39802025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
39812025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
39822025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
39832025-02-13T05:13:27.261ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
39842025-02-13T05:13:27.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 565 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60072 req_id = 1252b2fa-f5c2-4cf6-b775-3281a07428b7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
39852025-02-13T05:13:55.042ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:49830
39862025-02-13T05:13:55.042ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 34 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:49830 req_id = be6bfea6-0d34-4ab5-82e3-9ec1a13229cc response_code = 200 uri = /baseboard
39872025-02-13T05:13:55.177ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7226 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 06d90ee3-97e8-4438-8ab7-80d84d9efb10 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
39882025-02-13T05:13:56.502ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64751
39892025-02-13T05:13:56.502ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64751 req_id = 93d120c3-3a19-4c15-a030-3cdce2e49531 response_code = 200 uri = /vpc-routes
39902025-02-13T05:13:56.529ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36648
39912025-02-13T05:13:56.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36648 req_id = e363f166-08f6-49b1-bdb5-ea28aabb8a65 response_code = 200 uri = /vpc-routes
39922025-02-13T05:13:56.551ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
39932025-02-13T05:13:56.551ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64751 req_id = 64d84dad-41a1-4555-85fd-d5c75cc2f198 response_code = 204 uri = /eip-gateways
39942025-02-13T05:13:56.556ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33453
39952025-02-13T05:13:56.556ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33453 req_id = 389dc83c-8314-42b9-8598-c70cce0c3b74 response_code = 200 uri = /vpc-routes
39962025-02-13T05:13:56.562ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 396 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64751 req_id = 966f5bf2-068d-46d4-9cc1-d92a5cd0b5b3 response_code = 204 uri = /vpc-routes
39972025-02-13T05:13:56.574ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
39982025-02-13T05:13:56.574ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33453 req_id = f2e4edcf-ad4c-4f13-89c6-d885f5b805aa response_code = 204 uri = /eip-gateways
39992025-02-13T05:13:56.574ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34825
40002025-02-13T05:13:56.574ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40012025-02-13T05:13:56.619ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40022025-02-13T05:13:56.674ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 99600 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34825 req_id = 575eee60-0cd5-45fa-b03b-56e2bfd91813 response_code = 204 uri = /switch-ports
40032025-02-13T05:13:56.674ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48909
40042025-02-13T05:13:56.674ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55724
40052025-02-13T05:13:56.674ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44782
40062025-02-13T05:13:56.674ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56605
40072025-02-13T05:13:56.675ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
40082025-02-13T05:13:56.675ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40092025-02-13T05:13:56.675ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 377 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36648 req_id = fdf790b6-5819-4d03-ae52-8a10e834ab43 response_code = 204 uri = /eip-gateways
40102025-02-13T05:13:56.675ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 385 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34825 req_id = b290f421-ddd7-4485-96d6-1844bea9f322 response_code = 200 uri = /network-bootstore-config
40112025-02-13T05:13:56.679ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 4838 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44782 req_id = d2866a57-b727-4f6f-9bb9-a4b7ac93ef7d response_code = 200 uri = /v2p
40122025-02-13T05:13:56.679ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 4949 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48909 req_id = 8a1f42f0-01d2-453e-81ba-4e36698fc623 response_code = 200 uri = /v2p
40132025-02-13T05:13:56.680ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5021 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33453 req_id = bfecc173-180d-401f-b105-7a1385fd5db1 response_code = 204 uri = /vpc-routes
40142025-02-13T05:13:56.685ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 313 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36648 req_id = 70b37810-1834-43d8-ac64-0b0c1ab2c4e0 response_code = 204 uri = /vpc-routes
40152025-02-13T05:13:56.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52121
40162025-02-13T05:13:56.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52121 req_id = 98480d42-a5f7-41d8-9395-877da500fe7a response_code = 200 uri = /v2p
40172025-02-13T05:13:56.714ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40182025-02-13T05:13:56.770ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40192025-02-13T05:13:56.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 95933 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56605 req_id = 23dcce2c-5920-4430-ba24-078ab49f40f4 response_code = 204 uri = /switch-ports
40202025-02-13T05:13:56.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 333 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56605 req_id = ab9b31ca-5a05-4717-8717-0d57e912fe90 response_code = 200 uri = /network-bootstore-config
40212025-02-13T05:13:56.805ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40222025-02-13T05:13:56.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 178404 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:55724 req_id = 8325a993-a49f-4e2e-8da3-6a25a58d0b7d response_code = 204 uri = /switch-ports
40232025-02-13T05:13:56.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55724 req_id = 6dffcd90-8ee8-4cc5-b23f-c4f7a08dc55b response_code = 200 uri = /network-bootstore-config
40242025-02-13T05:14:25.073ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47502
40252025-02-13T05:14:25.074ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 37 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47502 req_id = 21330423-66da-43d5-9782-a518c8b8e5ac response_code = 200 uri = /baseboard
40262025-02-13T05:14:25.176ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 6796 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 9ba7bfa2-7d77-44c8-82b5-b227eb86a404 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
40272025-02-13T05:14:26.550ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60780
40282025-02-13T05:14:26.550ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60780 req_id = 205d7f72-0fcc-42c5-9648-b5e79c49130b response_code = 200 uri = /vpc-routes
40292025-02-13T05:14:26.630ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39071
40302025-02-13T05:14:26.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39071 req_id = e7bf514d-f9f7-42b3-8741-6a5600ff7d3d response_code = 200 uri = /vpc-routes
40312025-02-13T05:14:26.635ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42223
40322025-02-13T05:14:26.635ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42223 req_id = 8b1a46a3-364c-4888-9594-a41f2fd6eb2b response_code = 200 uri = /v2p
40332025-02-13T05:14:26.647ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
40342025-02-13T05:14:26.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39071 req_id = bc8f4f49-89d5-4e59-965a-273ee49f1611 response_code = 204 uri = /eip-gateways
40352025-02-13T05:14:26.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 369 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39071 req_id = 2b4934d0-4d76-431b-b97f-db49f01b794d response_code = 204 uri = /vpc-routes
40362025-02-13T05:14:26.685ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
40372025-02-13T05:14:26.685ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60780 req_id = e481a762-50fb-469d-a245-1e0077f0645c response_code = 204 uri = /eip-gateways
40382025-02-13T05:14:26.693ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60780 req_id = fbe5823b-65a1-44fb-8fbf-b9a2b04ea1e9 response_code = 204 uri = /vpc-routes
40392025-02-13T05:14:26.698ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64131
40402025-02-13T05:14:26.698ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64131 req_id = 0c885117-c155-4216-83c2-6b3003bc36f3 response_code = 200 uri = /v2p
40412025-02-13T05:14:26.727ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45652
40422025-02-13T05:14:26.727ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40432025-02-13T05:14:26.766ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40442025-02-13T05:14:26.829ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101578 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45652 req_id = 36ce3e99-3310-4e67-844d-bae36643bb47 response_code = 204 uri = /switch-ports
40452025-02-13T05:14:26.829ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33422
40462025-02-13T05:14:26.829ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53147
40472025-02-13T05:14:26.829ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53147 req_id = 41538201-1aab-4a23-bcd2-b7e02adf25ba response_code = 200 uri = /vpc-routes
40482025-02-13T05:14:26.829ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40492025-02-13T05:14:26.829ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 245 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45652 req_id = ccb0ec70-d85d-417a-8694-2aa571b93fe0 response_code = 200 uri = /network-bootstore-config
40502025-02-13T05:14:26.871ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40512025-02-13T05:14:26.875ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
40522025-02-13T05:14:26.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3714 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53147 req_id = af3132ed-8581-480d-8bdf-de05c6d1345a response_code = 204 uri = /eip-gateways
40532025-02-13T05:14:26.934ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 104815 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33422 req_id = 58889103-6ec5-4d1b-917e-e1201d9804ab response_code = 204 uri = /switch-ports
40542025-02-13T05:14:26.934ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33422 req_id = 4eb414cf-1ecb-4a78-aad2-539c7aff9bb7 response_code = 200 uri = /network-bootstore-config
40552025-02-13T05:14:26.966ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64653
40562025-02-13T05:14:26.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64653 req_id = 40bf8540-66b6-4e46-8256-0632bcc0b4a0 response_code = 200 uri = /v2p
40572025-02-13T05:14:26.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 248 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53147 req_id = 1089f89b-56be-450a-a07b-93334dc76833 response_code = 204 uri = /vpc-routes
40582025-02-13T05:14:27.030ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46820
40592025-02-13T05:14:27.030ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40602025-02-13T05:14:27.068ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40612025-02-13T05:14:27.123ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 93628 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46820 req_id = 318444aa-7a29-4acf-b554-4a52cc9f7921 response_code = 204 uri = /switch-ports
40622025-02-13T05:14:27.124ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46820 req_id = c350542b-091d-4fcc-ad4f-e80d9b926203 response_code = 200 uri = /network-bootstore-config
40632025-02-13T05:14:55.108ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41823
40642025-02-13T05:14:55.108ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 42 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41823 req_id = 334515eb-1833-46b3-b37d-ceec29de10fb response_code = 200 uri = /baseboard
40652025-02-13T05:14:55.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7008 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 4026fe44-5864-4ad9-8f84-a135218ad839 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
40662025-02-13T05:14:56.396ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35466
40672025-02-13T05:14:56.396ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35466 req_id = f6bf4c5e-b748-47e7-8627-cd3c96bbf701 response_code = 200 uri = /v2p
40682025-02-13T05:14:56.399ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41662
40692025-02-13T05:14:56.399ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41662 req_id = ff5ecf77-c492-49b7-be14-42a6b064f138 response_code = 200 uri = /vpc-routes
40702025-02-13T05:14:56.496ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
40712025-02-13T05:14:56.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41662 req_id = 2ddc6378-5f3d-4124-9795-5a2ea5051a15 response_code = 204 uri = /eip-gateways
40722025-02-13T05:14:56.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 310 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41662 req_id = a9da4c6e-a72c-426e-ab25-f7a9567caf3b response_code = 204 uri = /vpc-routes
40732025-02-13T05:14:56.591ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58057
40742025-02-13T05:14:56.591ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58057 req_id = e32ae759-71e1-43d2-a577-c79b2275c6b5 response_code = 200 uri = /vpc-routes
40752025-02-13T05:14:56.618ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35501
40762025-02-13T05:14:56.619ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35501 req_id = b8146cc9-5a12-44e6-bb96-b2e76d5419e4 response_code = 200 uri = /v2p
40772025-02-13T05:14:56.621ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50531
40782025-02-13T05:14:56.621ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40792025-02-13T05:14:56.668ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40802025-02-13T05:14:56.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 109768 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50531 req_id = 69b446dc-cd31-4121-a0b1-1b5602da6f50 response_code = 204 uri = /switch-ports
40812025-02-13T05:14:56.731ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44538
40822025-02-13T05:14:56.731ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45079
40832025-02-13T05:14:56.731ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40842025-02-13T05:14:56.731ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
40852025-02-13T05:14:56.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58057 req_id = 17526a71-1a88-4d6d-ae2a-cc89a7b70e9e response_code = 204 uri = /eip-gateways
40862025-02-13T05:14:56.735ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 3934 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45079 req_id = f3316be0-1d01-439c-870f-afbbced74a83 response_code = 200 uri = /vpc-routes
40872025-02-13T05:14:56.735ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 4126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50531 req_id = 6baf857e-1c64-4493-abbe-98f988c1146f response_code = 200 uri = /network-bootstore-config
40882025-02-13T05:14:56.737ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37884
40892025-02-13T05:14:56.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 355 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58057 req_id = ddafe551-776b-47b6-92bb-266f9a4a7d0c response_code = 204 uri = /vpc-routes
40902025-02-13T05:14:56.762ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49333
40912025-02-13T05:14:56.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 250 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49333 req_id = a1aeb12c-17ef-4acc-bcf7-1f85f5e666a9 response_code = 200 uri = /v2p
40922025-02-13T05:14:56.771ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
40932025-02-13T05:14:56.780ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
40942025-02-13T05:14:56.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45079 req_id = 0b9e3054-1ce1-4f2a-b551-d41e0cd096cc response_code = 204 uri = /eip-gateways
40952025-02-13T05:14:56.789ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 376 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45079 req_id = 75dbbd01-5310-4f7d-b08d-ff9f73a73511 response_code = 204 uri = /vpc-routes
40962025-02-13T05:14:56.828ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
40972025-02-13T05:14:56.828ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 96961 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44538 req_id = cd3f9810-dc2a-463c-80f3-b220dd2ea3c7 response_code = 204 uri = /switch-ports
40982025-02-13T05:14:56.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 756 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44538 req_id = b8332107-926a-4a48-ad26-6bcfd16e93c3 response_code = 200 uri = /network-bootstore-config
40992025-02-13T05:14:56.863ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
41002025-02-13T05:14:56.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 179992 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37884 req_id = 2567c2c0-6a90-495b-bcd3-99988900cd96 response_code = 204 uri = /switch-ports
41012025-02-13T05:14:56.918ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37884 req_id = d0d6b8f6-98d7-4a52-80fa-f8643a64a4f1 response_code = 200 uri = /network-bootstore-config
41022025-02-13T05:15:25.141ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:49820
41032025-02-13T05:15:25.141ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 36 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:49820 req_id = 08efa85c-3124-4570-b09b-7fde858970df response_code = 200 uri = /baseboard
41042025-02-13T05:15:25.176ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 8272 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 66432eb3-6eb3-42f5-ac76-d9dfced43f23 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
41052025-02-13T05:15:26.683ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51945
41062025-02-13T05:15:26.683ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51945 req_id = 079b1a82-bf6a-46ac-a456-7725561e9839 response_code = 200 uri = /vpc-routes
41072025-02-13T05:15:26.705ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41082025-02-13T05:15:26.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51945 req_id = 4076e8ae-d710-4855-b782-6b49e8870d54 response_code = 204 uri = /eip-gateways
41092025-02-13T05:15:26.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 318 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51945 req_id = a195ebac-76fd-4b52-a703-8f7185f3bf4d response_code = 204 uri = /vpc-routes
41102025-02-13T05:15:26.759ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34211
41112025-02-13T05:15:26.759ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34211 req_id = f5ab8ab6-7489-492e-a230-53719f9933e7 response_code = 200 uri = /vpc-routes
41122025-02-13T05:15:26.779ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62049
41132025-02-13T05:15:26.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62049 req_id = 9682c296-f9a5-4eb2-bfbf-14845664a280 response_code = 200 uri = /vpc-routes
41142025-02-13T05:15:26.823ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41152025-02-13T05:15:26.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34211 req_id = d399a3a8-fecf-4aea-863e-390a560f8abb response_code = 204 uri = /eip-gateways
41162025-02-13T05:15:26.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 312 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34211 req_id = e7922f45-d0a0-41b0-8ac9-ba2caf002800 response_code = 204 uri = /vpc-routes
41172025-02-13T05:15:26.839ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41182025-02-13T05:15:26.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62049 req_id = fe0b82fe-7ef6-47f9-a73c-fc29d92a4d18 response_code = 204 uri = /eip-gateways
41192025-02-13T05:15:26.846ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37185
41202025-02-13T05:15:26.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37185 req_id = 804b9f1f-d131-4833-af13-29d457a7f8e9 response_code = 200 uri = /v2p
41212025-02-13T05:15:26.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62049 req_id = 0ecfeaed-acdd-4e6f-b99e-c6483c288920 response_code = 204 uri = /vpc-routes
41222025-02-13T05:15:26.860ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37488
41232025-02-13T05:15:26.860ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
41242025-02-13T05:15:26.899ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
41252025-02-13T05:15:26.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101729 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37488 req_id = 4b975806-9fb4-4002-9da1-ccbd51fbc5e8 response_code = 204 uri = /switch-ports
41262025-02-13T05:15:26.962ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49397
41272025-02-13T05:15:26.962ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35470
41282025-02-13T05:15:26.962ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39871
41292025-02-13T05:15:26.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 217 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39871 req_id = 87ec243c-046d-437a-a5fc-8ddf963a6ea3 response_code = 200 uri = /v2p
41302025-02-13T05:15:26.962ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
41312025-02-13T05:15:26.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 298 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37488 req_id = d3b5d313-b2ed-4234-b5c2-0af2ab3cf04d response_code = 200 uri = /network-bootstore-config
41322025-02-13T05:15:27.002ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
41332025-02-13T05:15:27.018ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50194
41342025-02-13T05:15:27.018ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 210 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50194 req_id = 798a39f5-1642-4567-8821-6ae89ad821d4 response_code = 200 uri = /v2p
41352025-02-13T05:15:27.057ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
41362025-02-13T05:15:27.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 94840 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49397 req_id = 3d4036fc-5398-4014-8196-1c45e8858668 response_code = 204 uri = /switch-ports
41372025-02-13T05:15:27.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 367 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49397 req_id = df7beec2-7e2b-4eb0-a44b-4f12025ce891 response_code = 200 uri = /network-bootstore-config
41382025-02-13T05:15:27.095ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
41392025-02-13T05:15:27.154ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 191500 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35470 req_id = 6c3b9742-7540-46c1-ada0-7435aea524cc response_code = 204 uri = /switch-ports
41402025-02-13T05:15:27.154ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35470 req_id = e5dfff2d-15b3-4826-8ad8-ff1edd4e6f28 response_code = 200 uri = /network-bootstore-config
41412025-02-13T05:15:33.889ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:197
41422025-02-13T05:15:33.889ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
41432025-02-13T05:15:33.918ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1843, tv_nsec: 456257682 } time_to_next_cleanup = 599.99999548s
41442025-02-13T05:15:48.015ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
41452025-02-13T05:15:55.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 6829 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 1fec9474-3b72-4238-9a37-707766fa0b57 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
41462025-02-13T05:15:55.176ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50578
41472025-02-13T05:15:55.176ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 26 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50578 req_id = 481cdfe0-08f8-48aa-bcfb-08a33240aca4 response_code = 200 uri = /baseboard
41482025-02-13T05:15:56.478ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37828
41492025-02-13T05:15:56.479ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37828 req_id = cf65f9a2-022b-4067-bda5-b28c6bc444fc response_code = 200 uri = /vpc-routes
41502025-02-13T05:15:56.543ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64301
41512025-02-13T05:15:56.543ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64301 req_id = 8fa84fae-6da7-450e-975f-2b2bd494e78b response_code = 200 uri = /vpc-routes
41522025-02-13T05:15:56.553ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41532025-02-13T05:15:56.554ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37828 req_id = d10c88b7-8429-438d-8693-8fed97ba3269 response_code = 204 uri = /eip-gateways
41542025-02-13T05:15:56.566ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 339 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37828 req_id = 938be6c4-cd21-4b14-9ad9-2cfd915c88b3 response_code = 204 uri = /vpc-routes
41552025-02-13T05:15:56.601ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48132
41562025-02-13T05:15:56.601ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48132 req_id = a5f09627-3f77-4d2e-ae5a-5c8655dc5d4b response_code = 200 uri = /v2p
41572025-02-13T05:15:56.609ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41582025-02-13T05:15:56.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64301 req_id = c0692a37-219d-4efd-a6a5-9617d138ed1a response_code = 204 uri = /eip-gateways
41592025-02-13T05:15:56.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38859
41602025-02-13T05:15:56.609ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
41612025-02-13T05:15:56.664ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
41622025-02-13T05:15:56.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 129094 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38859 req_id = 803d6344-4575-4392-bc4c-f1711d29a5bb response_code = 204 uri = /switch-ports
41632025-02-13T05:15:56.739ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55184
41642025-02-13T05:15:56.739ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55096
41652025-02-13T05:15:56.739ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49243
41662025-02-13T05:15:56.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 415 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38859 req_id = 84a5b262-059e-4b20-979f-44e9bc061908 response_code = 200 uri = /network-bootstore-config
41672025-02-13T05:15:56.739ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
41682025-02-13T05:15:56.744ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49243 req_id = 0fc07ee6-e86c-489e-934d-4a1e3c98ebe0 response_code = 200 uri = /v2p
41692025-02-13T05:15:56.751ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 11628 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55096 req_id = 9cc1ca47-63c8-42a8-900c-dea9cf91373d response_code = 200 uri = /vpc-routes
41702025-02-13T05:15:56.751ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 12171 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64301 req_id = 5fd3ce1b-cf88-493b-a407-43ce8715ae8d response_code = 204 uri = /vpc-routes
41712025-02-13T05:15:56.765ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65184
41722025-02-13T05:15:56.786ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
41732025-02-13T05:15:56.810ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41742025-02-13T05:15:56.810ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 208 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55096 req_id = 473b7055-a022-46f9-bf5d-75e617affe74 response_code = 204 uri = /eip-gateways
41752025-02-13T05:15:56.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 541 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55096 req_id = fbabf42b-d8d3-4087-bf80-14dba3fef51b response_code = 204 uri = /vpc-routes
41762025-02-13T05:15:56.858ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
41772025-02-13T05:15:56.858ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 118602 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:55184 req_id = b984ccf6-4b46-4e05-b72e-ea141b14198b response_code = 204 uri = /switch-ports
41782025-02-13T05:15:56.863ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 469 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55184 req_id = a505e7cc-5c10-4541-99ef-d56c8413be0a response_code = 200 uri = /network-bootstore-config
41792025-02-13T05:15:56.906ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
41802025-02-13T05:15:56.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 206893 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:65184 req_id = 7ed69ac5-b500-4d8f-a438-edcf3b4a81a8 response_code = 204 uri = /switch-ports
41812025-02-13T05:15:56.973ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65184 req_id = 2fb1eb96-3d57-4339-9731-487adcbf66d7 response_code = 200 uri = /network-bootstore-config
41822025-02-13T05:15:57.008ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55859
41832025-02-13T05:15:57.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55859 req_id = 14baa61a-58a0-4568-95d6-ff6924c50880 response_code = 200 uri = /v2p
41842025-02-13T05:16:25.176ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7066 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = a80ee50e-6956-4bef-92b4-f0582905a07b response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
41852025-02-13T05:16:25.210ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36871
41862025-02-13T05:16:25.210ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 30 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36871 req_id = f0fd7f6b-b0e0-4e67-b978-d0427fe38766 response_code = 200 uri = /baseboard
41872025-02-13T05:16:26.480ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61502
41882025-02-13T05:16:26.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61502 req_id = 3153d470-98c3-4281-a3dd-f70d264e34da response_code = 200 uri = /vpc-routes
41892025-02-13T05:16:26.597ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41902025-02-13T05:16:26.597ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61502 req_id = a39a5c8c-03db-481f-9220-e70195db97b2 response_code = 204 uri = /eip-gateways
41912025-02-13T05:16:26.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 284 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61502 req_id = fda1e233-bd68-4868-82b5-d6132bd0c1d6 response_code = 204 uri = /vpc-routes
41922025-02-13T05:16:26.644ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54742
41932025-02-13T05:16:26.645ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54742 req_id = fcc3cb5a-8722-49d1-9ec8-a724ca165477 response_code = 200 uri = /vpc-routes
41942025-02-13T05:16:26.707ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
41952025-02-13T05:16:26.707ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54742 req_id = 29f603af-a5f1-415f-baf9-c40c9de878d9 response_code = 204 uri = /eip-gateways
41962025-02-13T05:16:26.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 312 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54742 req_id = 496a1c7a-9a58-4819-b791-e989df6b1447 response_code = 204 uri = /vpc-routes
41972025-02-13T05:16:26.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61430
41982025-02-13T05:16:26.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61430 req_id = 1a7146fd-dd1c-422b-909c-a6603b5c1b83 response_code = 200 uri = /vpc-routes
41992025-02-13T05:16:26.747ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
42002025-02-13T05:16:26.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61430 req_id = dd55e83d-a1d1-45e0-8693-ea9d6feed328 response_code = 204 uri = /eip-gateways
42012025-02-13T05:16:26.756ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 283 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61430 req_id = 72f21c64-e2d3-453f-82cd-0b518e4a937c response_code = 204 uri = /vpc-routes
42022025-02-13T05:16:26.760ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48037
42032025-02-13T05:16:26.760ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42042025-02-13T05:16:26.805ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42052025-02-13T05:16:26.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 104695 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48037 req_id = 87d83cf2-dc18-415d-9a6d-e68398a7b87f response_code = 204 uri = /switch-ports
42062025-02-13T05:16:26.865ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51015
42072025-02-13T05:16:26.865ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60765
42082025-02-13T05:16:26.865ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38411
42092025-02-13T05:16:26.865ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56252
42102025-02-13T05:16:26.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 255 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51015 req_id = 706604a4-9722-45ac-b4c4-fecf47802108 response_code = 200 uri = /v2p
42112025-02-13T05:16:26.865ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42122025-02-13T05:16:26.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 344 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48037 req_id = 8dead47e-73db-4308-b9a9-c09d13eac5a8 response_code = 200 uri = /network-bootstore-config
42132025-02-13T05:16:26.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 358 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56252 req_id = c3ede687-cfd6-487c-813a-d8c0d794bf9f response_code = 200 uri = /v2p
42142025-02-13T05:16:26.870ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 4266 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60765 req_id = c5b545f8-9851-4965-8b92-12b678e860da response_code = 200 uri = /v2p
42152025-02-13T05:16:26.876ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50671
42162025-02-13T05:16:26.905ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42172025-02-13T05:16:26.956ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42182025-02-13T05:16:26.956ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 90468 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38411 req_id = c570b7ad-e3ba-45c8-b763-7cf56895bace response_code = 204 uri = /switch-ports
42192025-02-13T05:16:26.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 580 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38411 req_id = 0afedc8e-804a-4f73-adff-d464bf7f0efd response_code = 200 uri = /network-bootstore-config
42202025-02-13T05:16:26.990ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42212025-02-13T05:16:27.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 167441 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50671 req_id = c5e8df5b-1ec5-48f8-9701-ea6a1ff1c105 response_code = 204 uri = /switch-ports
42222025-02-13T05:16:27.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50671 req_id = 53430309-9e6a-4cb1-bbc5-9d1ce57cbd1d response_code = 200 uri = /network-bootstore-config
42232025-02-13T05:16:55.176ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7968 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 4b8334a7-ee34-4f6f-bccb-df0b0bf25ea7 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
42242025-02-13T05:16:55.244ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46100
42252025-02-13T05:16:55.244ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 28 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46100 req_id = d10688bb-f639-4441-802b-3b875fdb3baf response_code = 200 uri = /baseboard
42262025-02-13T05:16:56.499ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45738
42272025-02-13T05:16:56.499ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45738 req_id = 58ef05bd-c87d-413d-b68a-3ea5d5e18367 response_code = 200 uri = /vpc-routes
42282025-02-13T05:16:56.501ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49169
42292025-02-13T05:16:56.501ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49169 req_id = 7e469396-d4d1-4ae6-b669-932f7157f3c0 response_code = 200 uri = /vpc-routes
42302025-02-13T05:16:56.529ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37810
42312025-02-13T05:16:56.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37810 req_id = 38a15158-a3fc-4fb5-80ba-4c85882d7df5 response_code = 200 uri = /vpc-routes
42322025-02-13T05:16:56.531ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
42332025-02-13T05:16:56.531ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49169 req_id = d058b8c5-41bc-4ecc-9406-28f7a12db091 response_code = 204 uri = /eip-gateways
42342025-02-13T05:16:56.543ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49169 req_id = d6638aee-b8f8-4a12-895e-09d7e8c7b1db response_code = 204 uri = /vpc-routes
42352025-02-13T05:16:56.569ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
42362025-02-13T05:16:56.569ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45738 req_id = 1edda2d4-90e5-4fc4-a255-da7b48b20823 response_code = 204 uri = /eip-gateways
42372025-02-13T05:16:56.580ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 337 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45738 req_id = dc8355ec-5036-48c0-a4ee-e2af54fb5d4c response_code = 204 uri = /vpc-routes
42382025-02-13T05:16:56.594ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46661
42392025-02-13T05:16:56.594ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42402025-02-13T05:16:56.643ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42412025-02-13T05:16:56.707ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 113321 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46661 req_id = a9147ab3-5233-4084-89ee-03aa705c754d response_code = 204 uri = /switch-ports
42422025-02-13T05:16:56.708ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60552
42432025-02-13T05:16:56.708ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
42442025-02-13T05:16:56.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37810 req_id = c4353e73-d719-45a2-9cd8-bab997230de3 response_code = 204 uri = /eip-gateways
42452025-02-13T05:16:56.708ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52176
42462025-02-13T05:16:56.708ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64436
42472025-02-13T05:16:56.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 230 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46661 req_id = e2cd94a3-66c6-49b7-acb7-0729cf2ec5cc response_code = 200 uri = /network-bootstore-config
42482025-02-13T05:16:56.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 194 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60552 req_id = b925fc29-2b53-4764-9cbf-4a657cd44903 response_code = 200 uri = /v2p
42492025-02-13T05:16:56.708ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42502025-02-13T05:16:56.714ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55969
42512025-02-13T05:16:56.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55969 req_id = e8e9b539-e10f-4b03-a358-dc0baab785b6 response_code = 200 uri = /v2p
42522025-02-13T05:16:56.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56276
42532025-02-13T05:16:56.719ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56276 req_id = f8fbe7c0-6263-45da-ad72-b184c9787ade response_code = 200 uri = /v2p
42542025-02-13T05:16:56.722ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 320 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37810 req_id = 0ad7780a-0ef6-4f59-a598-ae9a1d15d8a9 response_code = 204 uri = /vpc-routes
42552025-02-13T05:16:56.748ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42562025-02-13T05:16:56.800ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42572025-02-13T05:16:56.800ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 91494 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64436 req_id = 47a26dae-797d-4eb8-8354-98a5c0a22774 response_code = 204 uri = /switch-ports
42582025-02-13T05:16:56.804ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 681 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64436 req_id = 9b24e5b9-9598-4738-a6f0-63705f88c17b response_code = 200 uri = /network-bootstore-config
42592025-02-13T05:16:56.837ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42602025-02-13T05:16:56.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 183583 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52176 req_id = e707eaaa-25aa-4b29-abfe-0bcd38b86621 response_code = 204 uri = /switch-ports
42612025-02-13T05:16:56.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52176 req_id = 896d94c2-3866-4007-84ec-b98c17edaac9 response_code = 200 uri = /network-bootstore-config
42622025-02-13T05:17:25.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7260 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = c2f2c7ad-f078-41e1-9a90-f1f4ae824322 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
42632025-02-13T05:17:25.277ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42769
42642025-02-13T05:17:25.277ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 28 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42769 req_id = e2c7829a-abe8-4ca3-81a5-05f9742ee56c response_code = 200 uri = /baseboard
42652025-02-13T05:17:26.653ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51932
42662025-02-13T05:17:26.653ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51932 req_id = e614f8b3-fa42-46fd-acfe-1366d02bb90b response_code = 200 uri = /vpc-routes
42672025-02-13T05:17:26.705ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
42682025-02-13T05:17:26.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51932 req_id = 56430dcf-e0e4-48f0-b73d-a953ff5986ad response_code = 204 uri = /eip-gateways
42692025-02-13T05:17:26.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 245 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51932 req_id = 6f4c1527-e49d-41ef-8e53-76fdd86d08aa response_code = 204 uri = /vpc-routes
42702025-02-13T05:17:26.728ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40577
42712025-02-13T05:17:26.728ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40577 req_id = 83e2b12d-a833-4cc1-a69c-c631ad44b1fa response_code = 200 uri = /vpc-routes
42722025-02-13T05:17:26.744ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
42732025-02-13T05:17:26.744ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 44 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40577 req_id = 828cdcff-160c-46f9-b37b-750a4785a6e7 response_code = 204 uri = /eip-gateways
42742025-02-13T05:17:26.753ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40577 req_id = e583128e-898f-4c7c-95c5-64b7f75d5ac3 response_code = 204 uri = /vpc-routes
42752025-02-13T05:17:26.794ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61210
42762025-02-13T05:17:26.794ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61210 req_id = e6dc2dbe-c611-4ab5-b683-91a8986cad68 response_code = 200 uri = /v2p
42772025-02-13T05:17:26.820ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50052
42782025-02-13T05:17:26.820ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42792025-02-13T05:17:26.861ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42802025-02-13T05:17:26.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 96206 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50052 req_id = 42be23d9-36e1-4c57-bc45-82f59a0c755a response_code = 204 uri = /switch-ports
42812025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38242
42822025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43565
42832025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58201
42842025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38066
42852025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 335 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50052 req_id = a8a94dcd-a5f3-41a6-b8d2-2a5cde5cd17d response_code = 200 uri = /network-bootstore-config
42862025-02-13T05:17:26.917ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42872025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58201 req_id = 3226f132-327c-4c94-9110-1134eeb9ca87 response_code = 200 uri = /vpc-routes
42882025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 369 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38066 req_id = b371c0eb-57bc-4ffe-92db-2417393ed6d2 response_code = 200 uri = /v2p
42892025-02-13T05:17:26.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 404 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43565 req_id = 46c61214-3b86-4ba1-a7d7-7c73682c84da response_code = 200 uri = /v2p
42902025-02-13T05:17:26.939ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
42912025-02-13T05:17:26.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 656 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58201 req_id = fbd16447-037e-4cfe-a97e-7f51ac6387ac response_code = 204 uri = /eip-gateways
42922025-02-13T05:17:26.952ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1185 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58201 req_id = af983fa6-d0b6-46f9-9607-120ce91044b3 response_code = 204 uri = /vpc-routes
42932025-02-13T05:17:26.960ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42942025-02-13T05:17:26.981ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41759
42952025-02-13T05:17:27.023ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
42962025-02-13T05:17:27.023ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 105988 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38242 req_id = 8135ea2d-817e-4c61-8b90-01b1064817bd response_code = 204 uri = /switch-ports
42972025-02-13T05:17:27.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 652 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38242 req_id = bc2298ce-af7a-490c-b058-a88486de058e response_code = 200 uri = /network-bootstore-config
42982025-02-13T05:17:27.057ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
42992025-02-13T05:17:27.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 128871 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:41759 req_id = 3746c6be-592d-4fe6-825d-214cb5051e76 response_code = 204 uri = /switch-ports
43002025-02-13T05:17:27.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41759 req_id = 778b6b0e-3da9-4e3b-a773-51d0fed60ad3 response_code = 200 uri = /network-bootstore-config
43012025-02-13T05:17:55.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7096 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 3ec98d2d-df16-430b-9323-70f8bb95cb0c response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
43022025-02-13T05:17:55.313ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61558
43032025-02-13T05:17:55.313ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 33 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61558 req_id = b03ccef9-a059-40f8-857b-e0031bac6a98 response_code = 200 uri = /baseboard
43042025-02-13T05:17:56.473ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44583
43052025-02-13T05:17:56.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44583 req_id = db7194cb-1fa8-44eb-a038-dc45035bf509 response_code = 200 uri = /vpc-routes
43062025-02-13T05:17:56.523ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
43072025-02-13T05:17:56.524ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44583 req_id = 9c849e94-ea64-4218-b705-5df12652ca62 response_code = 204 uri = /eip-gateways
43082025-02-13T05:17:56.527ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62840
43092025-02-13T05:17:56.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62840 req_id = 4009a123-b4e9-4796-966f-38a0e7bf0fec response_code = 200 uri = /v2p
43102025-02-13T05:17:56.528ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37902
43112025-02-13T05:17:56.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37902 req_id = 44e56dfb-3507-4314-bfdb-5891228abf9f response_code = 200 uri = /vpc-routes
43122025-02-13T05:17:56.533ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44583 req_id = 9f510c9a-b6d6-4087-8260-1d3e834107b5 response_code = 204 uri = /vpc-routes
43132025-02-13T05:17:56.579ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
43142025-02-13T05:17:56.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37902 req_id = 3cbd0f30-53c9-40c8-8ef0-3e865ed2e93e response_code = 204 uri = /eip-gateways
43152025-02-13T05:17:56.585ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45961
43162025-02-13T05:17:56.585ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
43172025-02-13T05:17:56.627ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
43182025-02-13T05:17:56.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 107023 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45961 req_id = 1acf5afa-88b2-4331-b63d-6ae39d8a42c8 response_code = 204 uri = /switch-ports
43192025-02-13T05:17:56.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57571
43202025-02-13T05:17:56.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59126
43212025-02-13T05:17:56.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45961 req_id = 99609cf3-37b0-4153-b274-2d86616cd0a8 response_code = 200 uri = /network-bootstore-config
43222025-02-13T05:17:56.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 550 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37902 req_id = 9b70561f-f30d-4f19-bd0c-59fc39d2bde0 response_code = 204 uri = /vpc-routes
43232025-02-13T05:17:56.693ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
43242025-02-13T05:17:56.693ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59126 req_id = 828570b1-c80e-4504-9a0b-72bdaaea2fd7 response_code = 200 uri = /v2p
43252025-02-13T05:17:56.712ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47690
43262025-02-13T05:17:56.712ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47690 req_id = b58f357e-f99e-4fec-afcf-dd61994c4ab2 response_code = 200 uri = /vpc-routes
43272025-02-13T05:17:56.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41521
43282025-02-13T05:17:56.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41521 req_id = cef7fd4c-8951-46d1-9a1f-f783f459d1c6 response_code = 200 uri = /v2p
43292025-02-13T05:17:56.734ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
43302025-02-13T05:17:56.756ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
43312025-02-13T05:17:56.756ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47690 req_id = 84e52f13-95d6-4265-87ad-5d28679c08e2 response_code = 204 uri = /eip-gateways
43322025-02-13T05:17:56.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 370 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47690 req_id = 26b28955-49b6-4848-881a-fac10949fdf3 response_code = 204 uri = /vpc-routes
43332025-02-13T05:17:56.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 102680 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57571 req_id = 44d8a83e-73d6-4c5c-b80e-173c8db61a1d response_code = 204 uri = /switch-ports
43342025-02-13T05:17:56.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57571 req_id = 7df6dd29-d2c8-40e7-ba33-4bdce48fa4cb response_code = 200 uri = /network-bootstore-config
43352025-02-13T05:17:56.798ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33572
43362025-02-13T05:17:56.798ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
43372025-02-13T05:17:56.833ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
43382025-02-13T05:17:56.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 94073 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:33572 req_id = bb8d6a94-e114-480c-adeb-2bc7d94b8ff0 response_code = 204 uri = /switch-ports
43392025-02-13T05:17:56.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33572 req_id = 7c2bb9da-b393-42d6-8284-5042d6a98ce0 response_code = 200 uri = /network-bootstore-config
43402025-02-13T05:18:25.178ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 6955 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = d2368e52-6e9b-426e-afca-3e73919edb52 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
43412025-02-13T05:18:25.347ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35062
43422025-02-13T05:18:25.347ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 39 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35062 req_id = 370045d6-2413-4db9-87a6-1bb2c8b09f32 response_code = 200 uri = /baseboard
43432025-02-13T05:18:26.680ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37401
43442025-02-13T05:18:26.680ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37401 req_id = e25ae76a-45ac-44f1-a33c-dd184731d68c response_code = 200 uri = /vpc-routes
43452025-02-13T05:18:26.713ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54377
43462025-02-13T05:18:26.713ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
43472025-02-13T05:18:26.714ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 457 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54377 req_id = 4c1b4997-0256-422a-ba85-b76283bb950f response_code = 200 uri = /artifacts
43482025-02-13T05:18:26.749ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
43492025-02-13T05:18:26.749ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37401 req_id = 260faee0-1bbc-48b0-ae66-1d3db9f7ab5d response_code = 204 uri = /eip-gateways
43502025-02-13T05:18:26.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51595
43512025-02-13T05:18:26.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51595 req_id = 8fc77166-1a1c-43c1-8bb2-80d1b39463a8 response_code = 200 uri = /vpc-routes
43522025-02-13T05:18:26.763ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 334 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37401 req_id = 72ab25c9-c094-4612-92c8-9873ee759378 response_code = 204 uri = /vpc-routes
43532025-02-13T05:18:26.820ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62829
43542025-02-13T05:18:26.820ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
43552025-02-13T05:18:26.820ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62829 req_id = f2a579be-05a8-4ad6-b7d1-f5b9a5f0b669 response_code = 200 uri = /artifacts
43562025-02-13T05:18:26.823ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
43572025-02-13T05:18:26.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51595 req_id = 53fd672b-f4aa-4a3e-8909-43fd72997459 response_code = 204 uri = /eip-gateways
43582025-02-13T05:18:26.842ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48167
43592025-02-13T05:18:26.842ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48167 req_id = 3c16db61-31f7-42dc-a950-c350a0100e26 response_code = 200 uri = /v2p
43602025-02-13T05:18:26.864ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36529
43612025-02-13T05:18:26.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36529 req_id = ab1d141d-500f-4d36-8c53-de2b828031df response_code = 200 uri = /vpc-routes
43622025-02-13T05:18:26.878ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42158
43632025-02-13T05:18:26.878ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
43642025-02-13T05:18:26.923ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
43652025-02-13T05:18:26.988ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 109892 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42158 req_id = c70a9274-5561-4428-af2e-5e729c27f5de response_code = 204 uri = /switch-ports
43662025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45391
43672025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55530
43682025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43665
43692025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58057
43702025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57628
43712025-02-13T05:18:26.989ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
43722025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51938
43732025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 504 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42158 req_id = 31053de1-eb27-4cb2-87cd-005ab6d00bc8 response_code = 200 uri = /network-bootstore-config
43742025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 421 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36529 req_id = 65a179f1-61c1-4fb0-be07-0372e2b75361 response_code = 204 uri = /eip-gateways
43752025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59097
43762025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57154
43772025-02-13T05:18:26.989ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
43782025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 859 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51595 req_id = adc7dd0f-df9e-427b-904d-1b03459ebb99 response_code = 204 uri = /vpc-routes
43792025-02-13T05:18:26.989ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
43802025-02-13T05:18:26.989ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
43812025-02-13T05:18:26.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 451 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45391 req_id = 3b1da493-97d5-4b6c-927e-39f1b5335627 response_code = 200 uri = /v2p
43822025-02-13T05:18:26.989ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
43832025-02-13T05:18:26.990ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
43842025-02-13T05:18:26.990ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51938 req_id = 2fc34d32-dd66-43e3-842f-44bc21783316 response_code = 200 uri = /v2p
43852025-02-13T05:18:26.995ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
43862025-02-13T05:18:26.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 6833 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55530 req_id = 7d96f16b-7299-4c7e-b07c-87a135ad707b response_code = 200 uri = /artifacts
43872025-02-13T05:18:26.997ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
43882025-02-13T05:18:26.997ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
43892025-02-13T05:18:26.997ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
43902025-02-13T05:18:26.997ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
43912025-02-13T05:18:26.997ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
43922025-02-13T05:18:26.997ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
43932025-02-13T05:18:26.997ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
43942025-02-13T05:18:26.997ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
43952025-02-13T05:18:26.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 755 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58057 req_id = 5e14fdc1-c6f6-4eb5-a74a-2bb957d257bf response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
43962025-02-13T05:18:27.003ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1146 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36529 req_id = 66486c6a-e1d2-4f02-8db1-73912b6daf49 response_code = 204 uri = /vpc-routes
43972025-02-13T05:18:27.013ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
43982025-02-13T05:18:27.014ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
43992025-02-13T05:18:27.027ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
44002025-02-13T05:18:27.027ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
44012025-02-13T05:18:27.030ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
44022025-02-13T05:18:27.036ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
44032025-02-13T05:18:27.039ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44042025-02-13T05:18:27.039ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
44052025-02-13T05:18:27.052ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44062025-02-13T05:18:27.052ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
44072025-02-13T05:18:27.078ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44082025-02-13T05:18:27.079ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
44092025-02-13T05:18:27.093ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44102025-02-13T05:18:27.095ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
44112025-02-13T05:18:27.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36986
44122025-02-13T05:18:27.101ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
44132025-02-13T05:18:27.101ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 111717 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59097 req_id = a4b3cdcd-3ef0-49b9-8c28-a253489fbaec response_code = 204 uri = /switch-ports
44142025-02-13T05:18:27.105ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
44152025-02-13T05:18:27.105ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
44162025-02-13T05:18:27.105ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
44172025-02-13T05:18:27.105ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
44182025-02-13T05:18:27.105ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
44192025-02-13T05:18:27.105ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
44202025-02-13T05:18:27.105ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
44212025-02-13T05:18:27.105ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 4381 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36986 req_id = 7c8af659-f1d3-4c7e-b399-c8ac272c06f7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
44222025-02-13T05:18:27.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 484 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59097 req_id = 196b74a3-c1c6-4bac-809b-7eb78781fb30 response_code = 200 uri = /network-bootstore-config
44232025-02-13T05:18:27.110ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44242025-02-13T05:18:27.110ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
44252025-02-13T05:18:27.126ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44262025-02-13T05:18:27.126ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
44272025-02-13T05:18:27.142ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
44282025-02-13T05:18:27.150ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44292025-02-13T05:18:27.150ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
44302025-02-13T05:18:27.150ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43658
44312025-02-13T05:18:27.155ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:767 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b52770f8-08e7-4014-82c4-bd4251ffc77f, kind: Service { id: ec95727a-7af6-45d1-9962-81bf544db84b }, name: Name("external-dns-ec95727a-7af6-45d1-9962-81bf544db84b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 162, 76])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fc137de6-ec00-4a98-82db-f15d19cf43ba, kind: Service { id: f580e8f1-7812-496c-8421-463d5f7ea1f8 }, name: Name("external-dns-f580e8f1-7812-496c-8421-463d5f7ea1f8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 131, 247])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 32ffb45b-db9b-4edb-91ce-b8d647b82ef5, kind: Service { id: a8269f60-9104-4275-9edc-a31614d5c23e }, name: Name("nexus-a8269f60-9104-4275-9edc-a31614d5c23e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 143])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 86210250-964c-473a-b05f-690bc99588ac, kind: Service { id: f6817c28-f622-48d8-9425-69a8b0c4c59a }, name: Name("nexus-f6817c28-f622-48d8-9425-69a8b0c4c59a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9713236b-4ca0-45d6-803b-ee3989195d2b, kind: Service { id: 67cd1f2c-6dc2-422f-a2c8-74ebf346d228 }, name: Name("nexus-67cd1f2c-6dc2-422f-a2c8-74ebf346d228"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 123])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
44322025-02-13T05:18:27.155ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
44332025-02-13T05:18:27.155ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
44342025-02-13T05:18:27.155ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
44352025-02-13T05:18:27.155ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
44362025-02-13T05:18:27.155ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte0" rules = []
44372025-02-13T05:18:27.155ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:786 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
44382025-02-13T05:18:27.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 775 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43658 req_id = 8f7cc506-8ba0-453b-9fbb-583e977f0ace response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
44392025-02-13T05:18:27.164ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44402025-02-13T05:18:27.166ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
44412025-02-13T05:18:27.181ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44422025-02-13T05:18:27.182ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
44432025-02-13T05:18:27.197ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44442025-02-13T05:18:27.197ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
44452025-02-13T05:18:27.220ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 230476 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57154 req_id = 796cf11b-6853-4e72-a90c-b73f4f8b9d11 response_code = 204 uri = /switch-ports
44462025-02-13T05:18:27.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 334 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57154 req_id = 4b1abf9f-6337-4c5f-a1ea-34aadd36f202 response_code = 200 uri = /network-bootstore-config
44472025-02-13T05:18:27.223ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44482025-02-13T05:18:27.223ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
44492025-02-13T05:18:27.237ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44502025-02-13T05:18:27.237ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
44512025-02-13T05:18:27.254ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44522025-02-13T05:18:27.254ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
44532025-02-13T05:18:27.269ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44542025-02-13T05:18:27.269ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
44552025-02-13T05:18:27.287ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44562025-02-13T05:18:27.287ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
44572025-02-13T05:18:27.304ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44582025-02-13T05:18:27.304ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
44592025-02-13T05:18:27.322ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44602025-02-13T05:18:27.322ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
44612025-02-13T05:18:27.326ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 336905 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43665 req_id = 00c1d21f-1c77-435d-9950-f0e330adf084 response_code = 200 uri = /inventory
44622025-02-13T05:18:27.336ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 346682 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57628 req_id = 4fc6cbcc-84c2-4640-8991-87b600dc6ff0 response_code = 200 uri = /inventory
44632025-02-13T05:18:28.188ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54006
44642025-02-13T05:18:28.188ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
44652025-02-13T05:18:28.188ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
44662025-02-13T05:18:28.189ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/config/omicron-zones.json file = common/src/ledger.rs:177
44672025-02-13T05:18:28.189ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/config/omicron-zones.json file = common/src/ledger.rs:177
44682025-02-13T05:18:28.207ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 8e93081f-e6c7-48d4-8f2b-cc48a61e4a47 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44692025-02-13T05:18:28.207ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_8e93081f-e6c7-48d4-8f2b-cc48a61e4a47
44702025-02-13T05:18:28.245ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 92d599bb-7067-4926-9301-d5f5f983f111 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44712025-02-13T05:18:28.245ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_92d599bb-7067-4926-9301-d5f5f983f111
44722025-02-13T05:18:28.280ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 9488b08f-08d1-4217-90af-27a0f69b51f7 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44732025-02-13T05:18:28.280ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_9488b08f-08d1-4217-90af-27a0f69b51f7
44742025-02-13T05:18:28.302ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 762630b8-1a3b-4b4b-8ab5-c880e5bac6b1 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44752025-02-13T05:18:28.302ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_762630b8-1a3b-4b4b-8ab5-c880e5bac6b1
44762025-02-13T05:18:28.325ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 15d1957c-72d0-47b9-aa2b-993d1474e767 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44772025-02-13T05:18:28.325ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_15d1957c-72d0-47b9-aa2b-993d1474e767
44782025-02-13T05:18:28.346ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44792025-02-13T05:18:28.346ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a
44802025-02-13T05:18:28.369ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 888b0ace-3028-4fa4-a464-87268f90ca4e (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44812025-02-13T05:18:28.369ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_888b0ace-3028-4fa4-a464-87268f90ca4e
44822025-02-13T05:18:28.390ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 23605d2b-03bd-4254-b0b5-88ca6a94a43f (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44832025-02-13T05:18:28.390ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_23605d2b-03bd-4254-b0b5-88ca6a94a43f
44842025-02-13T05:18:28.411ZINFOSledAgent (StorageManager): Received DatasetsList { zpool: ZpoolName { id: 260eeacc-a97f-47c0-b604-100bb7d05b36 (zpool), kind: External }, tx: ... } file = sled-storage/src/manager.rs:607
44852025-02-13T05:18:28.412ZINFOSledAgent (StorageManager): Listing datasets within zpool file = sled-storage/src/manager.rs:1118 request = datasets_list zpool = oxp_260eeacc-a97f-47c0-b604-100bb7d05b36
44862025-02-13T05:18:28.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 236446 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54006 req_id = cc5c673a-e47c-4e7e-a23c-b8f9b429e15e response_code = 200 uri = /inventory
44872025-02-13T05:18:55.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 8347 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 41afc4b8-6d31-4871-866f-e25cfb6d43f2 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
44882025-02-13T05:18:55.380ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47463
44892025-02-13T05:18:55.380ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 29 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47463 req_id = 4a0d52b7-0134-4efe-9ede-bfcfca85d394 response_code = 200 uri = /baseboard
44902025-02-13T05:18:56.470ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51162
44912025-02-13T05:18:56.470ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51162 req_id = 804ce35b-1421-4c27-bfcd-ac52b3b8236d response_code = 200 uri = /vpc-routes
44922025-02-13T05:18:56.478ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38132
44932025-02-13T05:18:56.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38132 req_id = e7462d3e-b83e-4799-96c1-88813f179971 response_code = 200 uri = /vpc-routes
44942025-02-13T05:18:56.499ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
44952025-02-13T05:18:56.499ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38132 req_id = 6277bd3f-9629-44d5-89c0-c56d39b29325 response_code = 204 uri = /eip-gateways
44962025-02-13T05:18:56.519ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
44972025-02-13T05:18:56.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51162 req_id = 5366935d-10e7-4af1-aa8e-c7143c314dc9 response_code = 204 uri = /eip-gateways
44982025-02-13T05:18:56.530ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51162 req_id = 26082740-9acd-497d-b744-fa129d9df80a response_code = 204 uri = /vpc-routes
44992025-02-13T05:18:56.533ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35312
45002025-02-13T05:18:56.533ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35312 req_id = 98101eb4-3ece-49cc-ab91-641fdedfca64 response_code = 200 uri = /vpc-routes
45012025-02-13T05:18:56.552ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47290
45022025-02-13T05:18:56.552ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
45032025-02-13T05:18:56.604ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
45042025-02-13T05:18:56.681ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 129465 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47290 req_id = ed923d10-b650-4983-b5e5-edb9a0d60f6e response_code = 204 uri = /switch-ports
45052025-02-13T05:18:56.682ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57873
45062025-02-13T05:18:56.682ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43829
45072025-02-13T05:18:56.682ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
45082025-02-13T05:18:56.682ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 218 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35312 req_id = aea67468-0c05-4c23-bb28-1aa0199ca1c2 response_code = 204 uri = /eip-gateways
45092025-02-13T05:18:56.682ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 239 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57873 req_id = bc6e885b-db8c-48b3-a59b-86ced7ce4a40 response_code = 200 uri = /v2p
45102025-02-13T05:18:56.682ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
45112025-02-13T05:18:56.682ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 553 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38132 req_id = 3a5a3d49-a808-4e64-95f6-e89e10ddc500 response_code = 204 uri = /vpc-routes
45122025-02-13T05:18:56.682ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 386 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47290 req_id = 1d017bcb-f287-4df6-b5c4-aca9db622811 response_code = 200 uri = /network-bootstore-config
45132025-02-13T05:18:56.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50678
45142025-02-13T05:18:56.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 286 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35312 req_id = 2afb0b02-5d50-4ead-898d-41e1319af79c response_code = 204 uri = /vpc-routes
45152025-02-13T05:18:56.723ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64928
45162025-02-13T05:18:56.723ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64928 req_id = 0fd8597a-0ed2-413d-822e-2574f41663eb response_code = 200 uri = /v2p
45172025-02-13T05:18:56.724ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
45182025-02-13T05:18:56.776ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64973
45192025-02-13T05:18:56.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64973 req_id = 062fb271-657b-4540-a61b-24c4885f8324 response_code = 200 uri = /v2p
45202025-02-13T05:18:56.783ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
45212025-02-13T05:18:56.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 101334 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43829 req_id = e5c6561e-a17e-4ee9-a5c4-75748217ce62 response_code = 204 uri = /switch-ports
45222025-02-13T05:18:56.787ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 567 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43829 req_id = f9b24c5e-914e-4000-a6ca-d86f4f1ccde0 response_code = 200 uri = /network-bootstore-config
45232025-02-13T05:18:56.818ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
45242025-02-13T05:18:56.873ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 185246 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50678 req_id = 865c77b3-b8bf-42ca-b694-cd0ffcca1d54 response_code = 204 uri = /switch-ports
45252025-02-13T05:18:56.873ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 156 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50678 req_id = 94c006cd-40b3-40dc-b7b7-eda6795b684c response_code = 200 uri = /network-bootstore-config
45262025-02-13T05:19:07.312ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50344
45272025-02-13T05:19:07.312ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50344 req_id = e81fc02f-e20e-415d-a8a9-1d45f632702f response_code = 200 uri = /v2p
45282025-02-13T05:19:07.313ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:874 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(16186248) }
45292025-02-13T05:19:07.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 662 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50344 req_id = b06360c9-080f-4fab-824e-8ca124317935 response_code = 204 uri = /v2p
45302025-02-13T05:19:07.461ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59233
45312025-02-13T05:19:07.471ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:555 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(16186248))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [InstanceDisk { disk_id: 02146518-332b-4139-b2c2-c79fe1a63d9b, slot: 0, read_only: false, vcr_json: .., name: "disk-7db0f7fb154" }], boot_settings: Some(InstanceBootSettings { order: [02146518-332b-4139-b2c2-c79fe1a63d9b] }), cloud_init_bytes: Some(..) } instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 metadata = InstanceMetadata { silo_id: b8f13615-12e0-4837-bffe-48620fa7aaa2, project_id: 042a7604-b098-4fb4-8331-96be12e3c004 } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2025-02-13T05:19:07.461066109Z }
45322025-02-13T05:19:07.471ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:588 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 migration_id = None propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45332025-02-13T05:19:07.472ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1565 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 migration_id = None propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(16186248))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d4e0dc3f-8a80-40b2-beb9-6161c71d002e, kind: Instance { id: 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(16186248), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [InstanceDisk { disk_id: 02146518-332b-4139-b2c2-c79fe1a63d9b, slot: 0, read_only: false, vcr_json: .., name: "disk-7db0f7fb154" }], boot_settings: Some(InstanceBootSettings { order: [02146518-332b-4139-b2c2-c79fe1a63d9b] }), cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2025-02-13T05:19:07.461066109Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
45342025-02-13T05:19:07.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 12552 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59233 req_id = c4c3a6f2-ca6b-4043-ad98-2e88efdc2535 response_code = 200 uri = /vmms/783fce27-e546-4e3b-91f5-b68a22c014d6
45352025-02-13T05:19:07.564ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45652
45362025-02-13T05:19:07.566ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1643 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:45652 req_id = b44e17f3-7a62-4b46-8262-c3daf30c7d34 response_code = 200 uri = /vmms/783fce27-e546-4e3b-91f5-b68a22c014d6/state
45372025-02-13T05:19:07.567ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 16186248 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
45382025-02-13T05:19:07.567ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
45392025-02-13T05:19:07.609ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6 file = illumos-utils/src/zone.rs:325 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45402025-02-13T05:19:07.633ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6 file = illumos-utils/src/zone.rs:363 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45412025-02-13T05:19:07.655ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39598
45422025-02-13T05:19:07.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39598 req_id = d767ce78-3fd8-431a-b8a5-3ad4e5cab4cb response_code = 200 uri = /vpc-routes
45432025-02-13T05:19:07.673ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {d4e0dc3f-8a80-40b2-beb9-6161c71d002e: {10.151.2.175: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}, 10.151.2.176: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}}} }
45442025-02-13T05:19:09.251ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45452025-02-13T05:19:09.251ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6 zone file = illumos-utils/src/running_zone.rs:489 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45462025-02-13T05:19:09.446ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 34.12089ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45472025-02-13T05:19:09.490ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 66.976611ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45482025-02-13T05:19:09.573ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 275.666696ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45492025-02-13T05:19:09.861ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 355.645889ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45502025-02-13T05:19:10.230ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 1.140296226s file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45512025-02-13T05:19:11.384ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 1.08731383s file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45522025-02-13T05:19:12.482ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 864.648457ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45532025-02-13T05:19:13.359ZWARNSledAgent (InstanceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 947.440111ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 zone = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
45542025-02-13T05:19:14.335ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6 file = sled-agent/src/instance.rs:2073 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45552025-02-13T05:19:14.344ZWARNSledAgent (InstanceManager): wait for service svc:/system/illumos/propolis-server:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 43.742483ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45562025-02-13T05:19:14.398ZWARNSledAgent (InstanceManager): wait for service svc:/system/illumos/propolis-server:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 118.958078ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45572025-02-13T05:19:14.528ZWARNSledAgent (InstanceManager): wait for service svc:/system/illumos/propolis-server:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 258.247435ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45582025-02-13T05:19:14.799ZWARNSledAgent (InstanceManager): wait for service svc:/system/illumos/propolis-server:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 274.772663ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45592025-02-13T05:19:15.084ZWARNSledAgent (InstanceManager): wait for service svc:/system/illumos/propolis-server:default in zone Some("oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6") failed: Property not found. retry in 967.451443ms file = illumos-utils/src/svc.rs:42 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45602025-02-13T05:19:16.064ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:2082 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45612025-02-13T05:19:16.102ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:2106 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45622025-02-13T05:19:16.185ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:1216 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45632025-02-13T05:19:16.186ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1237 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
45642025-02-13T05:19:16.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 8514688 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39598 req_id = 01227be8-9de2-4fd1-bc43-744c2adc8a78 response_code = 204 uri = /eip-gateways
45652025-02-13T05:19:16.186ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2025-02-13T05:19:16.186876247Z } file = sled-agent/src/instance.rs:782 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45662025-02-13T05:19:16.186ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:804 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2025-02-13T05:19:16.186876247Z } propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45672025-02-13T05:19:16.186ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:701 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2025-02-13T05:19:16.186876247Z }, migration_in: None, migration_out: None }
45682025-02-13T05:19:16.278ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 295 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39598 req_id = 3c617c5d-dfaf-47ab-8c9a-c30aa62d72d2 response_code = 204 uri = /vpc-routes
45692025-02-13T05:19:16.290ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2025-02-13T05:19:16.290348710Z } file = sled-agent/src/instance.rs:782 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45702025-02-13T05:19:16.290ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:804 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-02-13T05:19:16.290348710Z } propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
45712025-02-13T05:19:16.290ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:701 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-02-13T05:19:16.290348710Z }, migration_in: None, migration_out: None }
45722025-02-13T05:19:25.173ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 7014 local_addr = [fd00:1122:3344:101::1]:40836 method = GET remote_addr = [fd00:1122:3344:101::d]:53516 req_id = 6e6f8bc5-d256-4f96-bbf6-055f4ef05942 response_code = 200 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 uri = /5835b48f-f45d-4fa8-91ed-38d933d73732
45732025-02-13T05:19:25.413ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46712
45742025-02-13T05:19:25.413ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 40 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46712 req_id = 2d690fda-1e10-42b2-8913-5e2401f95155 response_code = 200 uri = /baseboard
45752025-02-13T05:19:26.476ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34577
45762025-02-13T05:19:26.477ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 822 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34577 req_id = b3d34286-02b0-4086-bf9a-9bf5fa17907d response_code = 200 uri = /vmms/783fce27-e546-4e3b-91f5-b68a22c014d6/state
45772025-02-13T05:19:26.606ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59503
45782025-02-13T05:19:26.607ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59503 req_id = 09e3b406-a19a-4bf7-9625-fc47000701a5 response_code = 200 uri = /vmms/783fce27-e546-4e3b-91f5-b68a22c014d6/state
45792025-02-13T05:19:26.660ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63751
45802025-02-13T05:19:26.660ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63751 req_id = 011e90ee-86f3-49ae-a919-a0229699a952 response_code = 200 uri = /vmms/783fce27-e546-4e3b-91f5-b68a22c014d6/state
45812025-02-13T05:19:26.665ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50170
45822025-02-13T05:19:26.665ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50170 req_id = aef8244f-7ca4-48b8-a4fe-b4fde9f7175a response_code = 200 uri = /vpc-routes
45832025-02-13T05:19:26.696ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55557
45842025-02-13T05:19:26.696ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55557 req_id = a37a1062-3465-4ddc-87bc-28729079f03b response_code = 200 uri = /vpc-routes
45852025-02-13T05:19:26.728ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {d4e0dc3f-8a80-40b2-beb9-6161c71d002e: {10.151.2.175: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}, 10.151.2.176: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}}} }
45862025-02-13T05:19:26.728ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55557 req_id = c15928d5-51df-45cd-9faf-04380d2fb81d response_code = 204 uri = /eip-gateways
45872025-02-13T05:19:26.745ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {d4e0dc3f-8a80-40b2-beb9-6161c71d002e: {10.151.2.176: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}, 10.151.2.175: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}}} }
45882025-02-13T05:19:26.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50170 req_id = c4027a92-0d63-4d4d-ab75-546418be613d response_code = 204 uri = /eip-gateways
45892025-02-13T05:19:26.766ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 1045 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55557 req_id = 5fdf900c-2d11-4b3c-98ea-06ec1572e59c response_code = 204 uri = /vpc-routes
45902025-02-13T05:19:26.768ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50170 req_id = 61e18209-6d86-457d-bc30-f6d50f85bd41 response_code = 204 uri = /vpc-routes
45912025-02-13T05:19:26.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53772
45922025-02-13T05:19:26.795ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
45932025-02-13T05:19:26.840ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
45942025-02-13T05:19:26.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 106734 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53772 req_id = cb4dc876-91a7-4604-9068-8cfd252f3aed response_code = 204 uri = /switch-ports
45952025-02-13T05:19:26.902ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42978
45962025-02-13T05:19:26.902ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41381
45972025-02-13T05:19:26.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41381 req_id = b4da13c5-5e40-48e5-8271-26f943558c6a response_code = 200 uri = /vpc-routes
45982025-02-13T05:19:26.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53772 req_id = c7513c50-83ea-4b66-b4ed-93355be1f114 response_code = 200 uri = /network-bootstore-config
45992025-02-13T05:19:26.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42978 req_id = f368ef9f-11d5-4791-a7b0-3bf3b50ff33c response_code = 200 uri = /v2p
46002025-02-13T05:19:26.920ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {d4e0dc3f-8a80-40b2-beb9-6161c71d002e: {10.151.2.176: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}, 10.151.2.175: {f0a62aa8-3d88-4ee2-9a92-13e8d15b0d11}}} }
46012025-02-13T05:19:26.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41381 req_id = 2fd96c55-5214-4fbc-9cb1-6afeb189c240 response_code = 204 uri = /eip-gateways
46022025-02-13T05:19:26.929ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38597
46032025-02-13T05:19:26.929ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38597 req_id = c7bc853b-a8e6-42d0-9aee-a4d1c093ca3f response_code = 200 uri = /v2p
46042025-02-13T05:19:26.930ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40632
46052025-02-13T05:19:26.930ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
46062025-02-13T05:19:26.969ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
46072025-02-13T05:19:27.029ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 98900 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40632 req_id = f0a711fe-e290-4f75-8175-58d4d62dd16e response_code = 204 uri = /switch-ports
46082025-02-13T05:19:27.029ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60257
46092025-02-13T05:19:27.029ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48433
46102025-02-13T05:19:27.029ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58166
46112025-02-13T05:19:27.029ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 205 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40632 req_id = d828ad0f-c3e3-4a88-a6be-a170552bd689 response_code = 200 uri = /network-bootstore-config
46122025-02-13T05:19:27.029ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60257 req_id = 677c3426-99ea-4994-a0c8-66b435861060 response_code = 200 uri = /v2p
46132025-02-13T05:19:27.029ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4191
46142025-02-13T05:19:27.030ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 599 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41381 req_id = fa7c43c8-0b17-4d41-b498-298732f037f3 response_code = 204 uri = /vpc-routes
46152025-02-13T05:19:27.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 5324 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58166 req_id = d35aba76-4d25-4c1d-a25b-6516b5199113 response_code = 200 uri = /vmms/783fce27-e546-4e3b-91f5-b68a22c014d6/state
46162025-02-13T05:19:27.035ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2025-02-13T05:19:27.035548140Z } file = sled-agent/src/instance.rs:782 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46172025-02-13T05:19:27.035ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:804 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2025-02-13T05:19:27.035548140Z } propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46182025-02-13T05:19:27.035ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:701 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2025-02-13T05:19:27.035548140Z }, migration_in: None, migration_out: None }
46192025-02-13T05:19:27.069ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4215
46202025-02-13T05:19:27.132ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 102798 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48433 req_id = 3dbff3c9-6061-4cab-8b9c-dc48baed3c17 response_code = 204 uri = /switch-ports
46212025-02-13T05:19:27.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48433 req_id = 7f21ceda-acea-4e31-8718-874c712c6d46 response_code = 200 uri = /network-bootstore-config
46222025-02-13T05:19:27.137ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2025-02-13T05:19:27.137569872Z } file = sled-agent/src/instance.rs:782 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46232025-02-13T05:19:27.137ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:804 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2025-02-13T05:19:27.137569872Z } propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46242025-02-13T05:19:27.137ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:818 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46252025-02-13T05:19:27.137ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
46262025-02-13T05:19:27.137ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 12534411180434907110 n_samples = 6 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
46272025-02-13T05:19:27.138ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
46282025-02-13T05:19:27.138ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 17563785490459436117 n_samples = 6 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
46292025-02-13T05:19:27.138ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/af97f30c-e09d-4c57-b382-5397ac5562c4/debug/bundle/zone", "/pool/int/db71a81c-96cf-41f7-8c27-fbb635fcb5d5/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/8e93081f-e6c7-48d4-8f2b-cc48a61e4a47/crypt/debug", "/pool/ext/92d599bb-7067-4926-9301-d5f5f983f111/crypt/debug", "/pool/ext/9488b08f-08d1-4217-90af-27a0f69b51f7/crypt/debug", "/pool/ext/762630b8-1a3b-4b4b-8ab5-c880e5bac6b1/crypt/debug", "/pool/ext/15d1957c-72d0-47b9-aa2b-993d1474e767/crypt/debug", "/pool/ext/c0fb9e41-9277-4dbe-ba4b-2476f2db3e2a/crypt/debug", "/pool/ext/888b0ace-3028-4fa4-a464-87268f90ca4e/crypt/debug", "/pool/ext/23605d2b-03bd-4254-b0b5-88ca6a94a43f/crypt/debug", "/pool/ext/260eeacc-a97f-47c0-b604-100bb7d05b36/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6
46302025-02-13T05:19:27.788ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6", bundle_id: 4bf12944-605e-4035-aaad-9d475d8518b8 }, time_created: 2025-02-13T05:19:27.139456106Z, version: 0, cause: TerminatedInstance }
46312025-02-13T05:19:27.788ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_783fce27-e546-4e3b-91f5-b68a22c014d6 file = sled-agent/src/instance.rs:1316 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46322025-02-13T05:19:29.560ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:279 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46332025-02-13T05:19:29.577ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:1000 id = RouterId { vni: Vni(16186248), kind: System }
46342025-02-13T05:19:29.577ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:1000 id = RouterId { vni: Vni(16186248), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
46352025-02-13T05:19:29.578ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:701 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2025-02-13T05:19:29.578298670Z }, migration_in: None, migration_out: None }
46362025-02-13T05:19:29.676ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:701 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2025-02-13T05:19:29.578298670Z }, migration_in: None, migration_out: None }
46372025-02-13T05:19:29.681ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:1253 instance_id = 57ce81a4-40a7-4278-a40e-b11f33b0a5f4 propolis_id = 783fce27-e546-4e3b-91f5-b68a22c014d6
46382025-02-13T05:19:30.209ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53823
46392025-02-13T05:19:30.210ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53823 req_id = 41ea2594-3dac-44ad-9e5c-213d66314e2d response_code = 200 uri = /vpc-routes
46402025-02-13T05:19:30.255ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
46412025-02-13T05:19:30.255ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1237 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732
46422025-02-13T05:19:30.255ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53823 req_id = 7fee19ee-6339-4028-9179-0e1ac0d4c65f response_code = 204 uri = /eip-gateways
46432025-02-13T05:19:30.265ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35180
46442025-02-13T05:19:30.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35180 req_id = a5f5c1a0-7bd5-4041-819e-869cd09e7cab response_code = 200 uri = /v2p
46452025-02-13T05:19:30.268ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53823 req_id = d317dbdd-a4c7-4276-8ab5-18847c635aea response_code = 204 uri = /vpc-routes
46462025-02-13T05:19:31.693ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58804
46472025-02-13T05:19:31.693ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58804 req_id = d033759f-2cb5-470a-944d-7ce47961c06c response_code = 200 uri = /v2p
46482025-02-13T05:19:31.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33059
46492025-02-13T05:19:31.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33059 req_id = 7f8d9dff-9250-4b24-94b6-dfbe688d75c2 response_code = 200 uri = /vpc-routes
46502025-02-13T05:19:31.748ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
46512025-02-13T05:19:31.748ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33059 req_id = 968b9939-ab33-47b9-bfd5-4b68c8fb518e response_code = 204 uri = /eip-gateways
46522025-02-13T05:19:31.757ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33059 req_id = f99f3042-dde6-46fb-9962-918a73e6aee7 response_code = 204 uri = /vpc-routes
46532025-02-13T05:19:32.452ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:1025 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55749
46542025-02-13T05:19:32.452ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55749 req_id = 4bb6b272-5d18-472c-b180-b01549207f6b response_code = 200 uri = /vpc-routes
46552025-02-13T05:19:32.466ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1211 sled_id = 5835b48f-f45d-4fa8-91ed-38d933d73732 values = ExternalIpGatewayMap { mappings: {} }
46562025-02-13T05:19:32.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55749 req_id = d21225f6-e9e7-44c6-ad81-a84d9e7e1647 response_code = 204 uri = /eip-gateways
46572025-02-13T05:19:32.475ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:867 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55749 req_id = c1802e21-57d5-4e8b-b42f-75d21e1983f0 response_code = 204 uri = /vpc-routes