Max level shown:
1 [ Jan 11 01:17:32 Enabled. ]
2 [ Jan 11 01:17:32 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Jan 11 01:17:32 Method "start" exited with status 0. ]
4 [ Jan 11 01:17:32 Rereading configuration. ]
5 [ Jan 11 01:17:32 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72025-01-11T01:17:32.686ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82025-01-11T01:17:32.686ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92025-01-11T01:17:32.738ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102025-01-11T01:17:32.738ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112025-01-11T01:17:32.853ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122025-01-11T01:17:32.862ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132025-01-11T01:17:32.870ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142025-01-11T01:17:33.470ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:200::, width: 64 }
152025-01-11T01:17:33.613ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162025-01-11T01:17:33.613ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172025-01-11T01:17:33.614ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182025-01-11T01:17:33.614ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192025-01-11T01:17:33.614ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202025-01-11T01:17:33.614ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212025-01-11T01:17:33.617ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222025-01-11T01:17:33.617ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232025-01-11T01:17:33.617ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242025-01-11T01:17:33.617ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252025-01-11T01:17:33.617ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262025-01-11T01:17:33.617ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272025-01-11T01:17:33.617ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
282025-01-11T01:17:33.617ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292025-01-11T01:17:33.617ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302025-01-11T01:17:33.629ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312025-01-11T01:17:33.629ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool)
322025-01-11T01:17:33.730ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal }
332025-01-11T01:17:34.105ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:607
342025-01-11T01:17:34.105ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352025-01-11T01:17:34.105ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362025-01-11T01:17:34.105ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372025-01-11T01:17:34.105ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
382025-01-11T01:17:34.105ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392025-01-11T01:17:34.105ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402025-01-11T01:17:34.114ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412025-01-11T01:17:34.114ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 9404c118-e76c-41a3-b09c-850793e381f3 (zpool)
422025-01-11T01:17:34.201ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal }
432025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
482025-01-11T01:17:34.567ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
532025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
542025-01-11T01:17:34.567ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
602025-01-11T01:17:34.567ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
642025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
652025-01-11T01:17:34.567ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
662025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
722025-01-11T01:17:34.567ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
782025-01-11T01:17:34.567ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
812025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
822025-01-11T01:17:34.567ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842025-01-11T01:17:34.567ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852025-01-11T01:17:34.567ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862025-01-11T01:17:34.567ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872025-01-11T01:17:34.567ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
882025-01-11T01:17:34.568ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
892025-01-11T01:17:34.568ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
902025-01-11T01:17:34.568ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912025-01-11T01:17:34.568ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922025-01-11T01:17:34.568ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:607
932025-01-11T01:17:34.568ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
942025-01-11T01:17:34.568ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
952025-01-11T01:17:34.568ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962025-01-11T01:17:34.568ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972025-01-11T01:17:34.568ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982025-01-11T01:17:34.568ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992025-01-11T01:17:34.568ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002025-01-11T01:17:34.568ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012025-01-11T01:17:34.568ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022025-01-11T01:17:34.568ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1032025-01-11T01:17:34.568ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042025-01-11T01:17:34.568ZINFOSledAgent (bootstore): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052025-01-11T01:17:34.568ZINFOSledAgent (bootstore): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062025-01-11T01:17:34.568ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072025-01-11T01:17:34.568ZINFOSledAgent (bootstore): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082025-01-11T01:17:34.568ZINFOSledAgent (bootstore): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092025-01-11T01:17:34.568ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102025-01-11T01:17:34.568ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112025-01-11T01:17:34.568ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122025-01-11T01:17:34.578ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:748
1132025-01-11T01:17:34.578ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1127, tv_nsec: 353842733 } time_to_next_cleanup = 599.999920239s
1142025-01-11T01:17:34.579ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3973
1152025-01-11T01:17:34.579ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1162025-01-11T01:17:34.579ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/sled-agent-request.json file = common/src/ledger.rs:188
1172025-01-11T01:17:34.581ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/sled-agent-request.json file = common/src/ledger.rs:188
1182025-01-11T01:17:34.582ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fdb0:200::1]:80
1192025-01-11T01:17:34.583ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:200::1]:12346
1202025-01-11T01:17:34.584ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: Some( None, ), bgp_peers: [], autoneg: false, lldp: None, tx_eq: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1204
1212025-01-11T01:17:34.584ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1222025-01-11T01:17:34.584ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1232025-01-11T01:17:34.584ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1242025-01-11T01:17:34.584ZINFOSledAgent (RSS): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/rss-started.marker file = common/src/ledger.rs:188
1252025-01-11T01:17:34.584ZINFOSledAgent (RSS): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/rss-started.marker file = common/src/ledger.rs:188
1262025-01-11T01:17:34.584ZINFOSledAgent (RSS): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1272025-01-11T01:17:34.584ZINFOSledAgent (RSS): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1282025-01-11T01:17:34.584ZINFOSledAgent (RSS): RSS not previously run. Creating plans. file = sled-agent/src/rack_setup/service.rs:1253
1292025-01-11T01:17:34.584ZINFOSledAgent (RSS): Writing ledger to /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/.rss-started.marker.tmp file = common/src/ledger.rs:199
1302025-01-11T01:17:34.584ZINFOSledAgent (RSS): Writing ledger to /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/.rss-started.marker.tmp file = common/src/ledger.rs:199
1312025-01-11T01:17:34.585ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1285
1322025-01-11T01:17:34.585ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:66
1332025-01-11T01:17:34.585ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:200::1 file = sled-agent/src/rack_setup/plan/sled.rs:39
1342025-01-11T01:17:34.585ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1316
1352025-01-11T01:17:34.585ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1362025-01-11T01:17:34.585ZINFOSledAgent (bootstore): Writing ledger to /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1372025-01-11T01:17:34.585ZINFOSledAgent (bootstore): Writing ledger to /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1382025-01-11T01:17:34.585ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1392025-01-11T01:17:34.585ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: dfd15883-50bd-419a-82e1-b0ed2f66c5e6 (sled), rack_id: 90cda70f-4d85-4179-bfe4-b1e37ebf7366, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:200::1]:12346
1402025-01-11T01:17:34.585ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4270
1412025-01-11T01:17:34.585ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4664
1422025-01-11T01:17:34.590ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:200::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1432025-01-11T01:17:34.596ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:200::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1442025-01-11T01:17:34.596ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:200::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1452025-01-11T01:17:34.600ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462025-01-11T01:17:34.600ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472025-01-11T01:17:34.600ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:200::1]:42451
1482025-01-11T01:17:34.600ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: dfd15883-50bd-419a-82e1-b0ed2f66c5e6 (sled), rack_id: 90cda70f-4d85-4179-bfe4-b1e37ebf7366, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1492025-01-11T01:17:34.600ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1502025-01-11T01:17:34.600ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1512025-01-11T01:17:34.600ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:607
1522025-01-11T01:17:34.600ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1532025-01-11T01:17:34.600ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:813
1542025-01-11T01:17:34.600ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1552025-01-11T01:17:34.600ZINFOSledAgent (StorageManager): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1562025-01-11T01:17:34.600ZINFOSledAgent (StorageManager): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1572025-01-11T01:17:34.600ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:802
1582025-01-11T01:17:34.600ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:830
1592025-01-11T01:17:34.600ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:596
1602025-01-11T01:17:34.603ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1612025-01-11T01:17:34.603ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:701 result = DisksManagementResult { status: [] }
1622025-01-11T01:17:34.615ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1632025-01-11T01:17:34.636ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:325
1642025-01-11T01:17:34.653ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:363
1652025-01-11T01:17:34.684ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:412 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1662025-01-11T01:17:34.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
1672025-01-11T01:17:34.684ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1682025-01-11T01:17:34.684ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:424 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1692025-01-11T01:17:34.687ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 137.770748ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d196ae2b-7831-4e65-802a-f2868e8a9233", "content-length": "124", "date": "Sat, 11 Jan 2025 01:17:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d196ae2b-7831-4e65-802a-f2868e8a9233" } file = clients/ddm-admin-client/src/lib.rs:177
1702025-01-11T01:17:34.697ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_2026aef5-ecc1-428c-aac9-f90b84f99582/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1712025-01-11T01:17:34.721ZINFOSledAgent: successfully created encrypted zvol: name="oxi_2026aef5-ecc1-428c-aac9-f90b84f99582/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1722025-01-11T01:17:34.721ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_2026aef5-ecc1-428c-aac9-f90b84f99582/swap" file = sled-agent/src/swap_device.rs:96
1732025-01-11T01:17:34.726ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:439 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1742025-01-11T01:17:34.726ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:129
1752025-01-11T01:17:34.726ZINFOSledAgent (BackingFs): Ensuring dataset oxi_2026aef5-ecc1-428c-aac9-f90b84f99582/backing/fmd file = sled-agent/src/backing_fs.rs:139
1762025-01-11T01:17:34.789ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:174
1772025-01-11T01:17:34.826ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1782025-01-11T01:17:34.826ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 116.41878ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "53929db6-45fd-4e39-9346-1d49fd91d675", "content-length": "124", "date": "Sat, 11 Jan 2025 01:17:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "53929db6-45fd-4e39-9346-1d49fd91d675" } file = clients/ddm-admin-client/src/lib.rs:177
1792025-01-11T01:17:34.943ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1802025-01-11T01:17:34.944ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 88.664611ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d0d77627-397d-4a0c-b891-9ef37a6c54ec", "content-length": "124", "date": "Sat, 11 Jan 2025 01:17:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d0d77627-397d-4a0c-b891-9ef37a6c54ec" } file = clients/ddm-admin-client/src/lib.rs:177
1812025-01-11T01:17:35.009ZINFOSledAgent (BackingFs): Mounting oxi_2026aef5-ecc1-428c-aac9-f90b84f99582/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:182
1822025-01-11T01:17:35.021ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:190
1832025-01-11T01:17:35.021ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:190
1842025-01-11T01:17:35.021ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:190
1852025-01-11T01:17:35.021ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:196
1862025-01-11T01:17:35.033ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1872025-01-11T01:17:35.034ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 208.32102ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "44a03688-0f2b-4737-affb-379ca984aa65", "content-length": "124", "date": "Sat, 11 Jan 2025 01:17:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "44a03688-0f2b-4737-affb-379ca984aa65" } file = clients/ddm-admin-client/src/lib.rs:177
1882025-01-11T01:17:35.243ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892025-01-11T01:17:35.244ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 272.804831ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "07f3596b-101c-452c-898a-ee74a2e3dac7", "content-length": "124", "date": "Sat, 11 Jan 2025 01:17:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "07f3596b-101c-452c-898a-ee74a2e3dac7" } file = clients/ddm-admin-client/src/lib.rs:177
1902025-01-11T01:17:35.518ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912025-01-11T01:17:35.518ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 272.74824ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5f7a6678-b0d6-4982-a221-2a094d0f8e75", "content-length": "124", "date": "Sat, 11 Jan 2025 01:17:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5f7a6678-b0d6-4982-a221-2a094d0f8e75" } file = clients/ddm-admin-client/src/lib.rs:177
1922025-01-11T01:17:35.773ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1932025-01-11T01:17:35.779ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fd00:1122:3344:101::1]:48724 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1942025-01-11T01:17:35.779ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1952025-01-11T01:17:35.779ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:48724 file = oximeter/producer/src/lib.rs:279 interval = 30s producer_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1962025-01-11T01:17:35.780ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 152.599092ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1972025-01-11T01:17:35.781ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1982025-01-11T01:17:35.781ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
1992025-01-11T01:17:35.781ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:932 underlay_address = fd00:1122:3344:101::1
2002025-01-11T01:17:35.781ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
2012025-01-11T01:17:35.793ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2022025-01-11T01:17:35.934ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 488.560286ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2032025-01-11T01:17:36.424ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 542.94595ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2042025-01-11T01:17:36.969ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.060012825s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2052025-01-11T01:17:38.278ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2694
2062025-01-11T01:17:38.278ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2939
2072025-01-11T01:17:38.278ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2564
2082025-01-11T01:17:38.278ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2611
2092025-01-11T01:17:38.278ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:3020
2102025-01-11T01:17:38.278ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3065
2112025-01-11T01:17:38.280ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2864
2122025-01-11T01:17:38.282ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2605
2132025-01-11T01:17:38.282ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='7e37697c-4cff-42b2-a159-2bb576e979d2'/> <propval type="astring" name="rack_id" value='90cda70f-4d85-4179-bfe4-b1e37ebf7366'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:200::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:febb:473f'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:200::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="rack_id" value='90cda70f-4d85-4179-bfe4-b1e37ebf7366'/> <propval type="astring" name="sled_id" value='dfd15883-50bd-419a-82e1-b0ed2f66c5e6'/> <propval type="astring" name="sled_model" value='i86pc'/> <propval type="astring" name="sled_serial" value='sock'/> <propval type="astring" name="sled_revision" value='0'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='dfd15883-50bd-419a-82e1-b0ed2f66c5e6'/> <propval type="astring" name="rack_uuid" value='90cda70f-4d85-4179-bfe4-b1e37ebf7366'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='dfd15883-50bd-419a-82e1-b0ed2f66c5e6'/> <propval type="astring" name="rack_uuid" value='90cda70f-4d85-4179-bfe4-b1e37ebf7366'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2142025-01-11T01:17:38.282ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2152025-01-11T01:17:38.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 45.179923ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2162025-01-11T01:17:38.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 94.243242ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2172025-01-11T01:17:38.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 283.479922ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2182025-01-11T01:17:39.030ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.296214923s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2192025-01-11T01:17:39.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 350.572941ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202025-01-11T01:17:39.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.057520375s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212025-01-11T01:17:40.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.3815016s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222025-01-11T01:17:42.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.221295414s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232025-01-11T01:17:43.328ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 11.013286772s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2242025-01-11T01:17:44.311ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/nexus.rs:527 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2252025-01-11T01:17:44.340ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2262025-01-11T01:17:44.341ZINFOSledAgent (dropshot (Repo Depot)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fd00:1122:3344:101::1]:12348 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2272025-01-11T01:17:44.342ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fd00:1122:3344:101::1]:12345
2282025-01-11T01:17:44.342ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2292025-01-11T01:17:44.342ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2302025-01-11T01:17:44.342ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2312025-01-11T01:17:44.342ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2322025-01-11T01:17:44.342ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2332025-01-11T01:17:44.342ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: Some(None), bgp_peers: [], autoneg: false, lldp: None, tx_eq: None }], bgp: [], bfd: [] })))
2342025-01-11T01:17:44.342ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3973
2352025-01-11T01:17:44.342ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:200::1]:42451
2362025-01-11T01:17:44.543ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:200::1]:12346
2372025-01-11T01:17:44.543ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:200::1]:12346
2382025-01-11T01:17:44.548ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4294 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2392025-01-11T01:17:45.517ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2402025-01-11T01:17:45.519ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42613
2412025-01-11T01:17:45.519ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2422025-01-11T01:17:45.519ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
2432025-01-11T01:17:45.519ZINFOSledAgent (ServiceManager): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/omicron-zones.json file = common/src/ledger.rs:188
2442025-01-11T01:17:45.519ZINFOSledAgent (ServiceManager): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/omicron-zones.json file = common/src/ledger.rs:188
2452025-01-11T01:17:45.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 375 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42613 req_id = 4e36db09-2ffd-4438-b6eb-5b27fa234dfb response_code = 200 uri = /inventory
2462025-01-11T01:17:45.751ZINFOSledAgent (ServiceManager): refreshed zone-network-setup service with new configuration file = sled-agent/src/services.rs:4343
2472025-01-11T01:17:45.751ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4401
2482025-01-11T01:17:45.983ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2492025-01-11T01:17:45.983ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2502025-01-11T01:17:46.213ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2512025-01-11T01:17:46.228ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34307
2522025-01-11T01:17:46.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34307 req_id = 6627f099-e2f4-42d2-b514-cf924e97a96a response_code = 200 uri = /sled-role
2532025-01-11T01:17:46.259ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(2), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 69dbdd0a-897c-4e99-a470-a91142cf83fa (physical_disk), pool_id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: e2fde8f7-1205-4d60-ba0f-04707302a821 (physical_disk), pool_id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: a5a99cb6-3ab1-4115-b501-c7c97275d683 (physical_disk), pool_id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: e143a2be-1010-4476-9c0d-0e2241e91921 (physical_disk), pool_id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 27eb8077-ecfb-4b20-b5be-4d91c4c68586 (physical_disk), pool_id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: c8d3e277-d7d9-4b9c-a109-67853d99b0fe (physical_disk), pool_id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 1f22332d-7a1e-4714-b86c-21b6381791f9 (physical_disk), pool_id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 3a48a252-fa8d-4d14-a788-4b4abb5b2cd4 (physical_disk), pool_id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: e3ef064c-7782-4aab-9698-006236d975a9 (physical_disk), pool_id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:385 sled_address = [fd00:1122:3344:101::1]:12345
2542025-01-11T01:17:46.259ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43734
2552025-01-11T01:17:46.260ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:893 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2562025-01-11T01:17:46.260ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(2), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 69dbdd0a-897c-4e99-a470-a91142cf83fa (physical_disk), pool_id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: e2fde8f7-1205-4d60-ba0f-04707302a821 (physical_disk), pool_id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: a5a99cb6-3ab1-4115-b501-c7c97275d683 (physical_disk), pool_id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: e143a2be-1010-4476-9c0d-0e2241e91921 (physical_disk), pool_id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 27eb8077-ecfb-4b20-b5be-4d91c4c68586 (physical_disk), pool_id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: c8d3e277-d7d9-4b9c-a109-67853d99b0fe (physical_disk), pool_id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 1f22332d-7a1e-4714-b86c-21b6381791f9 (physical_disk), pool_id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 3a48a252-fa8d-4d14-a788-4b4abb5b2cd4 (physical_disk), pool_id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: e3ef064c-7782-4aab-9698-006236d975a9 (physical_disk), pool_id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:607
2572025-01-11T01:17:46.260ZINFOSledAgent (StorageManager): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2582025-01-11T01:17:46.260ZINFOSledAgent (StorageManager): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2592025-01-11T01:17:46.260ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:1298 request = omicron_physical_disks_ensure
2602025-01-11T01:17:46.260ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2612025-01-11T01:17:46.260ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2622025-01-11T01:17:46.260ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2632025-01-11T01:17:46.260ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2642025-01-11T01:17:46.270ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2652025-01-11T01:17:46.270ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool)
2662025-01-11T01:17:46.365ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }
2672025-01-11T01:17:46.374ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:239
2682025-01-11T01:17:46.374ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:241
2692025-01-11T01:17:46.374ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:245
2702025-01-11T01:17:46.374ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:247
2712025-01-11T01:17:46.374ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2722025-01-11T01:17:46.374ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2732025-01-11T01:17:46.374ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
2742025-01-11T01:17:46.408ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2752025-01-11T01:17:46.417ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone file = sled-storage/src/dataset.rs:302
2762025-01-11T01:17:46.519ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
2772025-01-11T01:17:46.552ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 file = sled-storage/src/dataset.rs:388
2782025-01-11T01:17:46.560ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2792025-01-11T01:17:46.560ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2802025-01-11T01:17:46.560ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2812025-01-11T01:17:46.560ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2822025-01-11T01:17:46.560ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2832025-01-11T01:17:46.570ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2842025-01-11T01:17:46.570ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool)
2852025-01-11T01:17:46.665ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }
2862025-01-11T01:17:46.674ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:239
2872025-01-11T01:17:46.674ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:241
2882025-01-11T01:17:46.674ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:245
2892025-01-11T01:17:46.674ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:247
2902025-01-11T01:17:46.674ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2912025-01-11T01:17:46.674ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2922025-01-11T01:17:46.674ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
2932025-01-11T01:17:46.709ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2942025-01-11T01:17:46.717ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone file = sled-storage/src/dataset.rs:302
2952025-01-11T01:17:46.855ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8d41776e-a84a-44c7-a848-bc443cc63109 file = sled-storage/src/dataset.rs:388
2962025-01-11T01:17:46.863ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
2972025-01-11T01:17:46.863ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
2982025-01-11T01:17:46.863ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
2992025-01-11T01:17:46.863ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3002025-01-11T01:17:46.863ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3012025-01-11T01:17:46.873ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3022025-01-11T01:17:46.873ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool)
3032025-01-11T01:17:46.968ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }
3042025-01-11T01:17:46.976ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:239
3052025-01-11T01:17:46.976ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:241
3062025-01-11T01:17:46.976ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:245
3072025-01-11T01:17:46.976ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:247
3082025-01-11T01:17:46.976ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3092025-01-11T01:17:46.976ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3102025-01-11T01:17:46.976ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3112025-01-11T01:17:47.011ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3122025-01-11T01:17:47.020ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone file = sled-storage/src/dataset.rs:302
3132025-01-11T01:17:47.162ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b file = sled-storage/src/dataset.rs:388
3142025-01-11T01:17:47.170ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3152025-01-11T01:17:47.170ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3162025-01-11T01:17:47.170ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3172025-01-11T01:17:47.170ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3182025-01-11T01:17:47.170ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3192025-01-11T01:17:47.180ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3202025-01-11T01:17:47.180ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool)
3212025-01-11T01:17:47.274ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }
3222025-01-11T01:17:47.283ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:239
3232025-01-11T01:17:47.283ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:241
3242025-01-11T01:17:47.283ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:245
3252025-01-11T01:17:47.283ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:247
3262025-01-11T01:17:47.283ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3272025-01-11T01:17:47.283ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3282025-01-11T01:17:47.283ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3292025-01-11T01:17:47.317ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3302025-01-11T01:17:47.326ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone file = sled-storage/src/dataset.rs:302
3312025-01-11T01:17:47.466ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1d1a100e-9784-4f32-981f-1a3200e989b4 file = sled-storage/src/dataset.rs:388
3322025-01-11T01:17:47.475ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3332025-01-11T01:17:47.475ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3342025-01-11T01:17:47.475ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3352025-01-11T01:17:47.475ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3362025-01-11T01:17:47.475ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3372025-01-11T01:17:47.484ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3382025-01-11T01:17:47.484ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = f5232c69-2469-4824-a4f8-901936b93f3d (zpool)
3392025-01-11T01:17:47.580ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }
3402025-01-11T01:17:47.588ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:239
3412025-01-11T01:17:47.588ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:241
3422025-01-11T01:17:47.588ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:245
3432025-01-11T01:17:47.588ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:247
3442025-01-11T01:17:47.588ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3452025-01-11T01:17:47.588ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3462025-01-11T01:17:47.588ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3472025-01-11T01:17:47.592ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
3482025-01-11T01:17:47.624ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3492025-01-11T01:17:47.632ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone file = sled-storage/src/dataset.rs:302
3502025-01-11T01:17:47.770ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f5232c69-2469-4824-a4f8-901936b93f3d file = sled-storage/src/dataset.rs:388
3512025-01-11T01:17:47.778ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3522025-01-11T01:17:47.778ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3532025-01-11T01:17:47.778ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3542025-01-11T01:17:47.778ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3552025-01-11T01:17:47.778ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3562025-01-11T01:17:47.788ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3572025-01-11T01:17:47.788ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool)
3582025-01-11T01:17:47.880ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }
3592025-01-11T01:17:47.888ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:239
3602025-01-11T01:17:47.888ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:241
3612025-01-11T01:17:47.888ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:245
3622025-01-11T01:17:47.888ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:247
3632025-01-11T01:17:47.888ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3642025-01-11T01:17:47.888ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3652025-01-11T01:17:47.888ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3662025-01-11T01:17:47.924ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3672025-01-11T01:17:47.932ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/zone file = sled-storage/src/dataset.rs:302
3682025-01-11T01:17:48.068ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_70afe259-ef7a-419a-97ed-388d46a1cb17 file = sled-storage/src/dataset.rs:388
3692025-01-11T01:17:48.076ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3702025-01-11T01:17:48.076ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3712025-01-11T01:17:48.076ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3722025-01-11T01:17:48.076ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3732025-01-11T01:17:48.076ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3742025-01-11T01:17:48.086ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3752025-01-11T01:17:48.086ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool)
3762025-01-11T01:17:48.185ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }
3772025-01-11T01:17:48.194ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:239
3782025-01-11T01:17:48.194ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:241
3792025-01-11T01:17:48.194ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:245
3802025-01-11T01:17:48.194ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:247
3812025-01-11T01:17:48.194ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3822025-01-11T01:17:48.194ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3832025-01-11T01:17:48.194ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
3842025-01-11T01:17:48.230ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3852025-01-11T01:17:48.238ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_0a1d8462-3db8-4b03-acdf-9503499e711d/crypt/zone file = sled-storage/src/dataset.rs:302
3862025-01-11T01:17:48.376ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_0a1d8462-3db8-4b03-acdf-9503499e711d file = sled-storage/src/dataset.rs:388
3872025-01-11T01:17:48.385ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3882025-01-11T01:17:48.385ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3892025-01-11T01:17:48.385ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3902025-01-11T01:17:48.385ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3912025-01-11T01:17:48.385ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3922025-01-11T01:17:48.395ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
3932025-01-11T01:17:48.395ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool)
3942025-01-11T01:17:48.490ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }
3952025-01-11T01:17:48.498ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:239
3962025-01-11T01:17:48.498ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:241
3972025-01-11T01:17:48.498ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:245
3982025-01-11T01:17:48.498ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:247
3992025-01-11T01:17:48.498ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4002025-01-11T01:17:48.498ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4012025-01-11T01:17:48.498ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
4022025-01-11T01:17:48.535ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4032025-01-11T01:17:48.543ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone file = sled-storage/src/dataset.rs:302
4042025-01-11T01:17:48.594ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4052025-01-11T01:17:48.689ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_094891fc-36ea-4f74-b7ba-91566beb4534 file = sled-storage/src/dataset.rs:388
4062025-01-11T01:17:48.697ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4072025-01-11T01:17:48.697ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4082025-01-11T01:17:48.697ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4092025-01-11T01:17:48.697ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4102025-01-11T01:17:48.697ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4112025-01-11T01:17:48.707ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4122025-01-11T01:17:48.708ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool)
4132025-01-11T01:17:48.804ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }
4142025-01-11T01:17:48.811ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:239
4152025-01-11T01:17:48.811ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:241
4162025-01-11T01:17:48.811ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:245
4172025-01-11T01:17:48.811ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:247
4182025-01-11T01:17:48.811ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4192025-01-11T01:17:48.812ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4202025-01-11T01:17:48.812ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:259
4212025-01-11T01:17:48.844ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4222025-01-11T01:17:48.852ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/zone file = sled-storage/src/dataset.rs:302
4232025-01-11T01:17:48.994ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a864efea-d3d9-4be0-84a5-f7e73206c783 file = sled-storage/src/dataset.rs:388
4242025-01-11T01:17:49.003ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4252025-01-11T01:17:49.003ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4262025-01-11T01:17:49.003ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4272025-01-11T01:17:49.003ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(2), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2026aef5-ecc1-428c-aac9-f90b84f99582 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9404c118-e76c-41a3-b09c-850793e381f3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4282025-01-11T01:17:49.003ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4292025-01-11T01:17:49.003ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4302025-01-11T01:17:49.003ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4312025-01-11T01:17:49.003ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:897 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4322025-01-11T01:17:49.003ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
4332025-01-11T01:17:49.003ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:923 generation = Generation(2) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4342025-01-11T01:17:49.003ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:928 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4352025-01-11T01:17:49.003ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:933 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4362025-01-11T01:17:49.003ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:122 gen = Generation(2) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4372025-01-11T01:17:49.003ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:939 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4382025-01-11T01:17:49.004ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:799 gen = Generation(2)
4392025-01-11T01:17:49.004ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:944 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4402025-01-11T01:17:49.004ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2744126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:43734 req_id = 41169f61-bf30-4488-8362-c6a25cbb0df6 response_code = 200 uri = /omicron-physical-disks
4412025-01-11T01:17:49.035ZINFOSledAgent (RSS): attempting to set up sled's datasets: DatasetsConfig { generation: Generation(1), datasets: {09b0c7ae-601d-42b3-8334-f0bae806ab74 (dataset): DatasetConfig { id: 09b0c7ae-601d-42b3-8334-f0bae806ab74 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_fa11dff5-c002-4f6a-8b1b-b726a1ca0a20" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0bfe0e66-2d27-46bc-8caf-06cd489a25b5 (dataset): DatasetConfig { id: 0bfe0e66-2d27-46bc-8caf-06cd489a25b5 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0fa1cc1d-5155-47bd-b94a-71f9b1fb9b84 (dataset): DatasetConfig { id: 0fa1cc1d-5155-47bd-b94a-71f9b1fb9b84 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 13e87a70-c620-488f-9c4c-52a3dd4c74b2 (dataset): DatasetConfig { id: 13e87a70-c620-488f-9c4c-52a3dd4c74b2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 175c6b4c-9871-49c4-9f6d-262d07281ddd (dataset): DatasetConfig { id: 175c6b4c-9871-49c4-9f6d-262d07281ddd (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1c1e6f02-6343-40d6-9459-0ad7ebbe72dd (dataset): DatasetConfig { id: 1c1e6f02-6343-40d6-9459-0ad7ebbe72dd (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1c709747-81e4-4696-a312-d8fc3952bf42 (dataset): DatasetConfig { id: 1c709747-81e4-4696-a312-d8fc3952bf42 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_oximeter_17bd3def-b7fb-4428-a2b5-5f1d745fd5a7" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 25ca6b38-5252-4b35-a88e-c539f69b81e0 (dataset): DatasetConfig { id: 25ca6b38-5252-4b35-a88e-c539f69b81e0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_9eee15bb-10a1-49cf-8087-610615834db2" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 291f0653-83e9-4b2e-b2ca-f378f3ee968a (dataset): DatasetConfig { id: 291f0653-83e9-4b2e-b2ca-f378f3ee968a (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2d702f49-c939-49fa-a71c-59908195b58e (dataset): DatasetConfig { id: 2d702f49-c939-49fa-a71c-59908195b58e (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_dbcc85a3-d91b-402e-bbe0-02359124c83e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 359fb39c-3752-4bd5-b0cf-d4273c8e9d01 (dataset): DatasetConfig { id: 359fb39c-3752-4bd5-b0cf-d4273c8e9d01 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 39a7645a-2aa7-401a-9d87-ca92258eea3b (dataset): DatasetConfig { id: 39a7645a-2aa7-401a-9d87-ca92258eea3b (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_ae3bb366-dd85-4558-a521-f33b9626f0b1" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3a11fb65-1a6f-46b5-91b2-28f3add03a1a (dataset): DatasetConfig { id: 3a11fb65-1a6f-46b5-91b2-28f3add03a1a (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3f086a5d-8479-4acf-9d3a-f3e97c63a17e (dataset): DatasetConfig { id: 3f086a5d-8479-4acf-9d3a-f3e97c63a17e (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 46a77ce2-fe19-4301-a3df-76f05cf64ad7 (dataset): DatasetConfig { id: 46a77ce2-fe19-4301-a3df-76f05cf64ad7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4b96cd1d-d3c9-483f-b15d-b2c6178ea9d2 (dataset): DatasetConfig { id: 4b96cd1d-d3c9-483f-b15d-b2c6178ea9d2 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_0278158b-ebb4-415c-bdce-d9833e816939" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4be48c53-1e9b-472d-b1dd-b0ea92506ee4 (dataset): DatasetConfig { id: 4be48c53-1e9b-472d-b1dd-b0ea92506ee4 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 4c1c3fe9-b50d-4e3d-bb08-75de0ef7b512 (dataset): DatasetConfig { id: 4c1c3fe9-b50d-4e3d-bb08-75de0ef7b512 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4c40936e-b09e-4620-a87b-09f3378f9031 (dataset): DatasetConfig { id: 4c40936e-b09e-4620-a87b-09f3378f9031 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_68e8e65f-6191-43af-b0e4-85cba11d9294" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4fd18126-78b3-4507-b435-97f1efede5d8 (dataset): DatasetConfig { id: 4fd18126-78b3-4507-b435-97f1efede5d8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5344f04e-8cc1-483e-818b-3f158258d437 (dataset): DatasetConfig { id: 5344f04e-8cc1-483e-818b-3f158258d437 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5c9f7970-3683-4463-9891-b44549db7a6e (dataset): DatasetConfig { id: 5c9f7970-3683-4463-9891-b44549db7a6e (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5cae68d6-bc26-4639-ad2d-229af4965de7 (dataset): DatasetConfig { id: 5cae68d6-bc26-4639-ad2d-229af4965de7 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_ca441e07-5182-40ca-bf73-5e03b6c846d9" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 695ba87e-8f5d-4571-bb5b-e28635a25d1d (dataset): DatasetConfig { id: 695ba87e-8f5d-4571-bb5b-e28635a25d1d (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6b882b89-3173-44ac-826e-c727dee18043 (dataset): DatasetConfig { id: 6b882b89-3173-44ac-826e-c727dee18043 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 71175310-7d13-498e-96d4-c720233cf99a (dataset): DatasetConfig { id: 71175310-7d13-498e-96d4-c720233cf99a (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 740c3ea5-134c-41a4-a0c2-f625e5be0130 (dataset): DatasetConfig { id: 740c3ea5-134c-41a4-a0c2-f625e5be0130 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 74439e02-29e6-43c9-8771-1221c3e33988 (dataset): DatasetConfig { id: 74439e02-29e6-43c9-8771-1221c3e33988 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 747bdce5-004e-4038-8931-6ce5eaa70188 (dataset): DatasetConfig { id: 747bdce5-004e-4038-8931-6ce5eaa70188 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 76242899-0849-4deb-a960-ce3e00a06ba8 (dataset): DatasetConfig { id: 76242899-0849-4deb-a960-ce3e00a06ba8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7739caad-267b-467b-8fd5-a50a0096bd4a (dataset): DatasetConfig { id: 7739caad-267b-467b-8fd5-a50a0096bd4a (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7ab33682-ac74-48a4-8e88-a54322dbce8c (dataset): DatasetConfig { id: 7ab33682-ac74-48a4-8e88-a54322dbce8c (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 808898e9-4271-442c-8793-5cc9c1376152 (dataset): DatasetConfig { id: 808898e9-4271-442c-8793-5cc9c1376152 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_95fbeccd-695b-4f6e-ae96-35054f4e2171" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 84345852-00ef-4077-a97e-21c8b3f9896f (dataset): DatasetConfig { id: 84345852-00ef-4077-a97e-21c8b3f9896f (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_00711fcf-5ea5-4dbf-918c-e21236b2cac3" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 86147b90-245f-4d55-baee-ea0bb169d87f (dataset): DatasetConfig { id: 86147b90-245f-4d55-baee-ea0bb169d87f (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 86a1286f-398c-4528-b313-5d2eb1b03de4 (dataset): DatasetConfig { id: 86a1286f-398c-4528-b313-5d2eb1b03de4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8701e2b8-78da-46c7-8957-25f526c725ec (dataset): DatasetConfig { id: 8701e2b8-78da-46c7-8957-25f526c725ec (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8bfdc6cb-8756-4ad3-a327-27f1df997f06 (dataset): DatasetConfig { id: 8bfdc6cb-8756-4ad3-a327-27f1df997f06 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_1877f8e9-a377-4960-91f4-09dd1c711f3b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8c405b05-f6c9-4658-b737-361075b5216f (dataset): DatasetConfig { id: 8c405b05-f6c9-4658-b737-361075b5216f (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8c69151f-038e-4a2b-8344-a8aae7b36024 (dataset): DatasetConfig { id: 8c69151f-038e-4a2b-8344-a8aae7b36024 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZone { name: "oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 935e6b13-1747-44c8-9f5a-1cb16f0f54a3 (dataset): DatasetConfig { id: 935e6b13-1747-44c8-9f5a-1cb16f0f54a3 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 9862a669-1af0-4cd7-a424-708337cd7a44 (dataset): DatasetConfig { id: 9862a669-1af0-4cd7-a424-708337cd7a44 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_dbbd09fb-c1a9-46dd-b9f4-782842395cbe" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 998e3b4e-5e5d-4e8c-ad7c-236c106bbd10 (dataset): DatasetConfig { id: 998e3b4e-5e5d-4e8c-ad7c-236c106bbd10 (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_38bf453c-46b6-4fd8-b901-1a35d301feef" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 9f6329a2-cbdc-4c02-b4da-15bb972ccf1a (dataset): DatasetConfig { id: 9f6329a2-cbdc-4c02-b4da-15bb972ccf1a (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b273b974-e613-4c89-a061-86dd9290f521 (dataset): DatasetConfig { id: b273b974-e613-4c89-a061-86dd9290f521 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b836dd39-6dd0-4b64-b902-cb810f581070 (dataset): DatasetConfig { id: b836dd39-6dd0-4b64-b902-cb810f581070 (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_d70ae7d1-a48f-47f5-8d1a-20559576d4aa" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, bce22923-b5c7-48f6-9398-4fb24e188e1f (dataset): DatasetConfig { id: bce22923-b5c7-48f6-9398-4fb24e188e1f (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_4ee703ec-7859-4e24-b277-ed30a36988ee" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, bf395ec1-38fc-4f51-a537-46e7b1d5ac27 (dataset): DatasetConfig { id: bf395ec1-38fc-4f51-a537-46e7b1d5ac27 (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c3ab36e1-bd7a-4bec-8aad-dd5ce5d60764 (dataset): DatasetConfig { id: c3ab36e1-bd7a-4bec-8aad-dd5ce5d60764 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_fb142081-54df-4946-b831-3eb44893e9e0" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c4519072-a3de-4af1-8d88-c36f09b720d6 (dataset): DatasetConfig { id: c4519072-a3de-4af1-8d88-c36f09b720d6 (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_027e6e69-94cf-42dd-a83a-77f1fe49eb61" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c92f5768-2e8d-43a9-bfc9-1d94e95d1092 (dataset): DatasetConfig { id: c92f5768-2e8d-43a9-bfc9-1d94e95d1092 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_9107f0ba-d732-48cb-866e-8ec55c9ba80a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, d135de29-ab84-42f7-96c8-de9249ed8ec5 (dataset): DatasetConfig { id: d135de29-ab84-42f7-96c8-de9249ed8ec5 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, dd142478-5fa8-4853-a153-e41c21a4801b (dataset): DatasetConfig { id: dd142478-5fa8-4853-a153-e41c21a4801b (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_cca5bf73-add8-4186-9f70-0867e2687634" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e0a1e35d-1416-42b2-aa9f-cb29a599ad0f (dataset): DatasetConfig { id: e0a1e35d-1416-42b2-aa9f-cb29a599ad0f (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_63191500-b9e6-45be-85a4-cdf6399d2f2a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e208e4f4-a40b-422f-9c8f-f1c622569309 (dataset): DatasetConfig { id: e208e4f4-a40b-422f-9c8f-f1c622569309 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_f84184b7-030d-4a37-95a3-1d6ee7166492" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e6cd7b55-e9f1-437d-958d-3b29fc69c698 (dataset): DatasetConfig { id: e6cd7b55-e9f1-437d-958d-3b29fc69c698 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e809dea7-039d-4b30-bc75-6586df4b6448 (dataset): DatasetConfig { id: e809dea7-039d-4b30-bc75-6586df4b6448 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_clickhouse_e5ee12e5-569a-47e9-b44f-6d568f6ce98c" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e81ec6aa-db05-4988-829a-241c09c7f9ac (dataset): DatasetConfig { id: e81ec6aa-db05-4988-829a-241c09c7f9ac (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, ecef64a2-124f-47bf-aa94-9f728aff9630 (dataset): DatasetConfig { id: ecef64a2-124f-47bf-aa94-9f728aff9630 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, eed4c031-32a7-4b3e-a74c-e9a65a249ba7 (dataset): DatasetConfig { id: eed4c031-32a7-4b3e-a74c-e9a65a249ba7 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_edb48777-436b-4f70-bed2-e6896c7ded23" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ef18a077-6232-49f8-9d2e-980ea45f1c27 (dataset): DatasetConfig { id: ef18a077-6232-49f8-9d2e-980ea45f1c27 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Clickhouse }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ef891673-559e-4357-b402-32dc2941fb9c (dataset): DatasetConfig { id: ef891673-559e-4357-b402-32dc2941fb9c (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f1cbcee8-bed9-4295-8534-2a65178413e7 (dataset): DatasetConfig { id: f1cbcee8-bed9-4295-8534-2a65178413e7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f9e89a5a-3f69-4e57-bca3-f8350c785e78 (dataset): DatasetConfig { id: f9e89a5a-3f69-4e57-bca3-f8350c785e78 (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, fb7cfc06-2178-4fbd-9d36-04a0bec5980c (dataset): DatasetConfig { id: fb7cfc06-2178-4fbd-9d36-04a0bec5980c (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_21413c29-505f-4c82-9c8b-50571ddaefa5" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fffddc0d-7712-4422-ad82-2e7d548c649a (dataset): DatasetConfig { id: fffddc0d-7712-4422-ad82-2e7d548c649a (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }} } file = sled-agent/src/rack_setup/service.rs:460 sled_address = [fd00:1122:3344:101::1]:12345
4422025-01-11T01:17:49.035ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51820
4432025-01-11T01:17:49.036ZINFOSledAgent: datasets ensure file = sled-agent/src/sled_agent.rs:859 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4442025-01-11T01:17:49.036ZINFOSledAgent (StorageManager): Received DatasetsEnsure { config: DatasetsConfig { generation: Generation(1), datasets: {09b0c7ae-601d-42b3-8334-f0bae806ab74 (dataset): DatasetConfig { id: 09b0c7ae-601d-42b3-8334-f0bae806ab74 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_fa11dff5-c002-4f6a-8b1b-b726a1ca0a20" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0bfe0e66-2d27-46bc-8caf-06cd489a25b5 (dataset): DatasetConfig { id: 0bfe0e66-2d27-46bc-8caf-06cd489a25b5 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0fa1cc1d-5155-47bd-b94a-71f9b1fb9b84 (dataset): DatasetConfig { id: 0fa1cc1d-5155-47bd-b94a-71f9b1fb9b84 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 13e87a70-c620-488f-9c4c-52a3dd4c74b2 (dataset): DatasetConfig { id: 13e87a70-c620-488f-9c4c-52a3dd4c74b2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 175c6b4c-9871-49c4-9f6d-262d07281ddd (dataset): DatasetConfig { id: 175c6b4c-9871-49c4-9f6d-262d07281ddd (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1c1e6f02-6343-40d6-9459-0ad7ebbe72dd (dataset): DatasetConfig { id: 1c1e6f02-6343-40d6-9459-0ad7ebbe72dd (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1c709747-81e4-4696-a312-d8fc3952bf42 (dataset): DatasetConfig { id: 1c709747-81e4-4696-a312-d8fc3952bf42 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_oximeter_17bd3def-b7fb-4428-a2b5-5f1d745fd5a7" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 25ca6b38-5252-4b35-a88e-c539f69b81e0 (dataset): DatasetConfig { id: 25ca6b38-5252-4b35-a88e-c539f69b81e0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_9eee15bb-10a1-49cf-8087-610615834db2" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 291f0653-83e9-4b2e-b2ca-f378f3ee968a (dataset): DatasetConfig { id: 291f0653-83e9-4b2e-b2ca-f378f3ee968a (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2d702f49-c939-49fa-a71c-59908195b58e (dataset): DatasetConfig { id: 2d702f49-c939-49fa-a71c-59908195b58e (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_dbcc85a3-d91b-402e-bbe0-02359124c83e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 359fb39c-3752-4bd5-b0cf-d4273c8e9d01 (dataset): DatasetConfig { id: 359fb39c-3752-4bd5-b0cf-d4273c8e9d01 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 39a7645a-2aa7-401a-9d87-ca92258eea3b (dataset): DatasetConfig { id: 39a7645a-2aa7-401a-9d87-ca92258eea3b (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_ae3bb366-dd85-4558-a521-f33b9626f0b1" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3a11fb65-1a6f-46b5-91b2-28f3add03a1a (dataset): DatasetConfig { id: 3a11fb65-1a6f-46b5-91b2-28f3add03a1a (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3f086a5d-8479-4acf-9d3a-f3e97c63a17e (dataset): DatasetConfig { id: 3f086a5d-8479-4acf-9d3a-f3e97c63a17e (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 46a77ce2-fe19-4301-a3df-76f05cf64ad7 (dataset): DatasetConfig { id: 46a77ce2-fe19-4301-a3df-76f05cf64ad7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4b96cd1d-d3c9-483f-b15d-b2c6178ea9d2 (dataset): DatasetConfig { id: 4b96cd1d-d3c9-483f-b15d-b2c6178ea9d2 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_0278158b-ebb4-415c-bdce-d9833e816939" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4be48c53-1e9b-472d-b1dd-b0ea92506ee4 (dataset): DatasetConfig { id: 4be48c53-1e9b-472d-b1dd-b0ea92506ee4 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 4c1c3fe9-b50d-4e3d-bb08-75de0ef7b512 (dataset): DatasetConfig { id: 4c1c3fe9-b50d-4e3d-bb08-75de0ef7b512 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4c40936e-b09e-4620-a87b-09f3378f9031 (dataset): DatasetConfig { id: 4c40936e-b09e-4620-a87b-09f3378f9031 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_68e8e65f-6191-43af-b0e4-85cba11d9294" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4fd18126-78b3-4507-b435-97f1efede5d8 (dataset): DatasetConfig { id: 4fd18126-78b3-4507-b435-97f1efede5d8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5344f04e-8cc1-483e-818b-3f158258d437 (dataset): DatasetConfig { id: 5344f04e-8cc1-483e-818b-3f158258d437 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5c9f7970-3683-4463-9891-b44549db7a6e (dataset): DatasetConfig { id: 5c9f7970-3683-4463-9891-b44549db7a6e (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5cae68d6-bc26-4639-ad2d-229af4965de7 (dataset): DatasetConfig { id: 5cae68d6-bc26-4639-ad2d-229af4965de7 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_ca441e07-5182-40ca-bf73-5e03b6c846d9" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 695ba87e-8f5d-4571-bb5b-e28635a25d1d (dataset): DatasetConfig { id: 695ba87e-8f5d-4571-bb5b-e28635a25d1d (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6b882b89-3173-44ac-826e-c727dee18043 (dataset): DatasetConfig { id: 6b882b89-3173-44ac-826e-c727dee18043 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 71175310-7d13-498e-96d4-c720233cf99a (dataset): DatasetConfig { id: 71175310-7d13-498e-96d4-c720233cf99a (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 740c3ea5-134c-41a4-a0c2-f625e5be0130 (dataset): DatasetConfig { id: 740c3ea5-134c-41a4-a0c2-f625e5be0130 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 74439e02-29e6-43c9-8771-1221c3e33988 (dataset): DatasetConfig { id: 74439e02-29e6-43c9-8771-1221c3e33988 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 747bdce5-004e-4038-8931-6ce5eaa70188 (dataset): DatasetConfig { id: 747bdce5-004e-4038-8931-6ce5eaa70188 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 76242899-0849-4deb-a960-ce3e00a06ba8 (dataset): DatasetConfig { id: 76242899-0849-4deb-a960-ce3e00a06ba8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7739caad-267b-467b-8fd5-a50a0096bd4a (dataset): DatasetConfig { id: 7739caad-267b-467b-8fd5-a50a0096bd4a (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7ab33682-ac74-48a4-8e88-a54322dbce8c (dataset): DatasetConfig { id: 7ab33682-ac74-48a4-8e88-a54322dbce8c (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 808898e9-4271-442c-8793-5cc9c1376152 (dataset): DatasetConfig { id: 808898e9-4271-442c-8793-5cc9c1376152 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_95fbeccd-695b-4f6e-ae96-35054f4e2171" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 84345852-00ef-4077-a97e-21c8b3f9896f (dataset): DatasetConfig { id: 84345852-00ef-4077-a97e-21c8b3f9896f (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_00711fcf-5ea5-4dbf-918c-e21236b2cac3" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 86147b90-245f-4d55-baee-ea0bb169d87f (dataset): DatasetConfig { id: 86147b90-245f-4d55-baee-ea0bb169d87f (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 86a1286f-398c-4528-b313-5d2eb1b03de4 (dataset): DatasetConfig { id: 86a1286f-398c-4528-b313-5d2eb1b03de4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8701e2b8-78da-46c7-8957-25f526c725ec (dataset): DatasetConfig { id: 8701e2b8-78da-46c7-8957-25f526c725ec (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8bfdc6cb-8756-4ad3-a327-27f1df997f06 (dataset): DatasetConfig { id: 8bfdc6cb-8756-4ad3-a327-27f1df997f06 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_1877f8e9-a377-4960-91f4-09dd1c711f3b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8c405b05-f6c9-4658-b737-361075b5216f (dataset): DatasetConfig { id: 8c405b05-f6c9-4658-b737-361075b5216f (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8c69151f-038e-4a2b-8344-a8aae7b36024 (dataset): DatasetConfig { id: 8c69151f-038e-4a2b-8344-a8aae7b36024 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: TransientZone { name: "oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 935e6b13-1747-44c8-9f5a-1cb16f0f54a3 (dataset): DatasetConfig { id: 935e6b13-1747-44c8-9f5a-1cb16f0f54a3 (dataset), name: DatasetName { pool_name: ZpoolName { id: a864efea-d3d9-4be0-84a5-f7e73206c783 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 9862a669-1af0-4cd7-a424-708337cd7a44 (dataset): DatasetConfig { id: 9862a669-1af0-4cd7-a424-708337cd7a44 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_dbbd09fb-c1a9-46dd-b9f4-782842395cbe" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 998e3b4e-5e5d-4e8c-ad7c-236c106bbd10 (dataset): DatasetConfig { id: 998e3b4e-5e5d-4e8c-ad7c-236c106bbd10 (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_38bf453c-46b6-4fd8-b901-1a35d301feef" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 9f6329a2-cbdc-4c02-b4da-15bb972ccf1a (dataset): DatasetConfig { id: 9f6329a2-cbdc-4c02-b4da-15bb972ccf1a (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b273b974-e613-4c89-a061-86dd9290f521 (dataset): DatasetConfig { id: b273b974-e613-4c89-a061-86dd9290f521 (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b836dd39-6dd0-4b64-b902-cb810f581070 (dataset): DatasetConfig { id: b836dd39-6dd0-4b64-b902-cb810f581070 (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_d70ae7d1-a48f-47f5-8d1a-20559576d4aa" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, bce22923-b5c7-48f6-9398-4fb24e188e1f (dataset): DatasetConfig { id: bce22923-b5c7-48f6-9398-4fb24e188e1f (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_4ee703ec-7859-4e24-b277-ed30a36988ee" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, bf395ec1-38fc-4f51-a537-46e7b1d5ac27 (dataset): DatasetConfig { id: bf395ec1-38fc-4f51-a537-46e7b1d5ac27 (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c3ab36e1-bd7a-4bec-8aad-dd5ce5d60764 (dataset): DatasetConfig { id: c3ab36e1-bd7a-4bec-8aad-dd5ce5d60764 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_fb142081-54df-4946-b831-3eb44893e9e0" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c4519072-a3de-4af1-8d88-c36f09b720d6 (dataset): DatasetConfig { id: c4519072-a3de-4af1-8d88-c36f09b720d6 (dataset), name: DatasetName { pool_name: ZpoolName { id: 70afe259-ef7a-419a-97ed-388d46a1cb17 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_027e6e69-94cf-42dd-a83a-77f1fe49eb61" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c92f5768-2e8d-43a9-bfc9-1d94e95d1092 (dataset): DatasetConfig { id: c92f5768-2e8d-43a9-bfc9-1d94e95d1092 (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_9107f0ba-d732-48cb-866e-8ec55c9ba80a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, d135de29-ab84-42f7-96c8-de9249ed8ec5 (dataset): DatasetConfig { id: d135de29-ab84-42f7-96c8-de9249ed8ec5 (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, dd142478-5fa8-4853-a153-e41c21a4801b (dataset): DatasetConfig { id: dd142478-5fa8-4853-a153-e41c21a4801b (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_cca5bf73-add8-4186-9f70-0867e2687634" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e0a1e35d-1416-42b2-aa9f-cb29a599ad0f (dataset): DatasetConfig { id: e0a1e35d-1416-42b2-aa9f-cb29a599ad0f (dataset), name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_63191500-b9e6-45be-85a4-cdf6399d2f2a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e208e4f4-a40b-422f-9c8f-f1c622569309 (dataset): DatasetConfig { id: e208e4f4-a40b-422f-9c8f-f1c622569309 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_f84184b7-030d-4a37-95a3-1d6ee7166492" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e6cd7b55-e9f1-437d-958d-3b29fc69c698 (dataset): DatasetConfig { id: e6cd7b55-e9f1-437d-958d-3b29fc69c698 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e809dea7-039d-4b30-bc75-6586df4b6448 (dataset): DatasetConfig { id: e809dea7-039d-4b30-bc75-6586df4b6448 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_clickhouse_e5ee12e5-569a-47e9-b44f-6d568f6ce98c" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e81ec6aa-db05-4988-829a-241c09c7f9ac (dataset): DatasetConfig { id: e81ec6aa-db05-4988-829a-241c09c7f9ac (dataset), name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, ecef64a2-124f-47bf-aa94-9f728aff9630 (dataset): DatasetConfig { id: ecef64a2-124f-47bf-aa94-9f728aff9630 (dataset), name: DatasetName { pool_name: ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, eed4c031-32a7-4b3e-a74c-e9a65a249ba7 (dataset): DatasetConfig { id: eed4c031-32a7-4b3e-a74c-e9a65a249ba7 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_edb48777-436b-4f70-bed2-e6896c7ded23" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ef18a077-6232-49f8-9d2e-980ea45f1c27 (dataset): DatasetConfig { id: ef18a077-6232-49f8-9d2e-980ea45f1c27 (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: Clickhouse }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ef891673-559e-4357-b402-32dc2941fb9c (dataset): DatasetConfig { id: ef891673-559e-4357-b402-32dc2941fb9c (dataset), name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f1cbcee8-bed9-4295-8534-2a65178413e7 (dataset): DatasetConfig { id: f1cbcee8-bed9-4295-8534-2a65178413e7 (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, f9e89a5a-3f69-4e57-bca3-f8350c785e78 (dataset): DatasetConfig { id: f9e89a5a-3f69-4e57-bca3-f8350c785e78 (dataset), name: DatasetName { pool_name: ZpoolName { id: 0a1d8462-3db8-4b03-acdf-9503499e711d (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, fb7cfc06-2178-4fbd-9d36-04a0bec5980c (dataset): DatasetConfig { id: fb7cfc06-2178-4fbd-9d36-04a0bec5980c (dataset), name: DatasetName { pool_name: ZpoolName { id: 094891fc-36ea-4f74-b7ba-91566beb4534 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_21413c29-505f-4c82-9c8b-50571ddaefa5" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fffddc0d-7712-4422-ad82-2e7d548c649a (dataset): DatasetConfig { id: fffddc0d-7712-4422-ad82-2e7d548c649a (dataset), name: DatasetName { pool_name: ZpoolName { id: 1d1a100e-9784-4f32-981f-1a3200e989b4 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }} }, tx: ... } file = sled-storage/src/manager.rs:607
4452025-01-11T01:17:49.036ZINFOSledAgent (StorageManager): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/omicron-datasets.json file = common/src/ledger.rs:188 request = datasets_ensure
4462025-01-11T01:17:49.036ZINFOSledAgent (StorageManager): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/omicron-datasets.json file = common/src/ledger.rs:188 request = datasets_ensure
4472025-01-11T01:17:49.036ZINFOSledAgent (StorageManager): No previously-stored 'requested datasets', creating new ledger file = sled-storage/src/manager.rs:909 request = datasets_ensure
4482025-01-11T01:17:49.051ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4492025-01-11T01:17:49.051ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4502025-01-11T01:17:49.051ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone/oxz_cockroachdb_fa11dff5-c002-4f6a-8b1b-b726a1ca0a20"), full_name: "oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone/oxz_cockroachdb_fa11dff5-c002-4f6a-8b1b-b726a1ca0a20" } file = sled-storage/src/manager.rs:1431
4512025-01-11T01:17:49.114ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4522025-01-11T01:17:49.114ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4532025-01-11T01:17:49.114ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/internal_dns" } file = sled-storage/src/manager.rs:1431
4542025-01-11T01:17:49.164ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4552025-01-11T01:17:49.164ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4562025-01-11T01:17:49.164ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/094891fc-36ea-4f74-b7ba-91566beb4534/crypt/debug"), full_name: "oxp_094891fc-36ea-4f74-b7ba-91566beb4534/crypt/debug" } file = sled-storage/src/manager.rs:1431
4572025-01-11T01:17:49.196ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4582025-01-11T01:17:49.196ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4592025-01-11T01:17:49.196ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crucible" } file = sled-storage/src/manager.rs:1431
4602025-01-11T01:17:49.246ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4612025-01-11T01:17:49.246ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4622025-01-11T01:17:49.246ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/internal_dns" } file = sled-storage/src/manager.rs:1431
4632025-01-11T01:17:49.294ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4642025-01-11T01:17:49.294ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4652025-01-11T01:17:49.294ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone" } file = sled-storage/src/manager.rs:1431
4662025-01-11T01:17:49.323ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4672025-01-11T01:17:49.323ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4682025-01-11T01:17:49.323ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_oximeter_17bd3def-b7fb-4428-a2b5-5f1d745fd5a7"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_oximeter_17bd3def-b7fb-4428-a2b5-5f1d745fd5a7" } file = sled-storage/src/manager.rs:1431
4692025-01-11T01:17:49.380ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4702025-01-11T01:17:49.380ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4712025-01-11T01:17:49.380ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone/oxz_cockroachdb_9eee15bb-10a1-49cf-8087-610615834db2"), full_name: "oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone/oxz_cockroachdb_9eee15bb-10a1-49cf-8087-610615834db2" } file = sled-storage/src/manager.rs:1431
4722025-01-11T01:17:49.442ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4732025-01-11T01:17:49.442ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4742025-01-11T01:17:49.442ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone/oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone/oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e" } file = sled-storage/src/manager.rs:1431
4752025-01-11T01:17:49.501ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4762025-01-11T01:17:49.501ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4772025-01-11T01:17:49.501ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_cockroachdb_dbcc85a3-d91b-402e-bbe0-02359124c83e"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_cockroachdb_dbcc85a3-d91b-402e-bbe0-02359124c83e" } file = sled-storage/src/manager.rs:1431
4782025-01-11T01:17:49.561ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4792025-01-11T01:17:49.561ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4802025-01-11T01:17:49.561ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
4812025-01-11T01:17:49.597ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
4822025-01-11T01:17:49.612ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4832025-01-11T01:17:49.612ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4842025-01-11T01:17:49.612ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/zone/oxz_crucible_ae3bb366-dd85-4558-a521-f33b9626f0b1"), full_name: "oxp_70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/zone/oxz_crucible_ae3bb366-dd85-4558-a521-f33b9626f0b1" } file = sled-storage/src/manager.rs:1431
4852025-01-11T01:17:49.673ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4862025-01-11T01:17:49.673ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4872025-01-11T01:17:49.673ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f" } file = sled-storage/src/manager.rs:1431
4882025-01-11T01:17:49.732ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4892025-01-11T01:17:49.732ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4902025-01-11T01:17:49.732ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/debug"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/debug" } file = sled-storage/src/manager.rs:1431
4912025-01-11T01:17:49.761ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4922025-01-11T01:17:49.761ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4932025-01-11T01:17:49.761ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_0a1d8462-3db8-4b03-acdf-9503499e711d/crucible" } file = sled-storage/src/manager.rs:1431
4942025-01-11T01:17:49.811ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4952025-01-11T01:17:49.811ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
4962025-01-11T01:17:49.811ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_crucible_0278158b-ebb4-415c-bdce-d9833e816939"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_crucible_0278158b-ebb4-415c-bdce-d9833e816939" } file = sled-storage/src/manager.rs:1431
4972025-01-11T01:17:49.871ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
4982025-01-11T01:17:49.872ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
4992025-01-11T01:17:49.872ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/debug"), full_name: "oxp_a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/debug" } file = sled-storage/src/manager.rs:1431
5002025-01-11T01:17:49.901ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5012025-01-11T01:17:49.901ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5022025-01-11T01:17:49.901ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_094891fc-36ea-4f74-b7ba-91566beb4534/crucible" } file = sled-storage/src/manager.rs:1431
5032025-01-11T01:17:49.945ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5042025-01-11T01:17:49.945ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5052025-01-11T01:17:49.945ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/zone/oxz_nexus_68e8e65f-6191-43af-b0e4-85cba11d9294"), full_name: "oxp_a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/zone/oxz_nexus_68e8e65f-6191-43af-b0e4-85cba11d9294" } file = sled-storage/src/manager.rs:1431
5062025-01-11T01:17:50.006ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5072025-01-11T01:17:50.006ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5082025-01-11T01:17:50.006ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
5092025-01-11T01:17:50.056ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5102025-01-11T01:17:50.056ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5112025-01-11T01:17:50.056ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone" } file = sled-storage/src/manager.rs:1431
5122025-01-11T01:17:50.086ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5132025-01-11T01:17:50.086ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5142025-01-11T01:17:50.086ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone"), full_name: "oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone" } file = sled-storage/src/manager.rs:1431
5152025-01-11T01:17:50.114ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5162025-01-11T01:17:50.114ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5172025-01-11T01:17:50.114ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/zone/oxz_crucible_ca441e07-5182-40ca-bf73-5e03b6c846d9"), full_name: "oxp_a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/zone/oxz_crucible_ca441e07-5182-40ca-bf73-5e03b6c846d9" } file = sled-storage/src/manager.rs:1431
5182025-01-11T01:17:50.174ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5192025-01-11T01:17:50.174ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5202025-01-11T01:17:50.174ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crucible" } file = sled-storage/src/manager.rs:1431
5212025-01-11T01:17:50.224ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5222025-01-11T01:17:50.224ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5232025-01-11T01:17:50.224ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
5242025-01-11T01:17:50.272ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5252025-01-11T01:17:50.272ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5262025-01-11T01:17:50.272ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/debug"), full_name: "oxp_70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/debug" } file = sled-storage/src/manager.rs:1431
5272025-01-11T01:17:50.299ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5282025-01-11T01:17:50.299ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5292025-01-11T01:17:50.299ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/external_dns" } file = sled-storage/src/manager.rs:1431
5302025-01-11T01:17:50.346ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5312025-01-11T01:17:50.346ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5322025-01-11T01:17:50.346ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/internal_dns" } file = sled-storage/src/manager.rs:1431
5332025-01-11T01:17:50.394ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5342025-01-11T01:17:50.394ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5352025-01-11T01:17:50.394ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/external_dns" } file = sled-storage/src/manager.rs:1431
5362025-01-11T01:17:50.443ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5372025-01-11T01:17:50.443ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5382025-01-11T01:17:50.443ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crucible" } file = sled-storage/src/manager.rs:1431
5392025-01-11T01:17:50.489ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5402025-01-11T01:17:50.489ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5412025-01-11T01:17:50.489ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
5422025-01-11T01:17:50.535ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5432025-01-11T01:17:50.535ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5442025-01-11T01:17:50.535ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/f5232c69-2469-4824-a4f8-901936b93f3d/crypt/debug"), full_name: "oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crypt/debug" } file = sled-storage/src/manager.rs:1431
5452025-01-11T01:17:50.564ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5462025-01-11T01:17:50.564ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5472025-01-11T01:17:50.564ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone/oxz_crucible_95fbeccd-695b-4f6e-ae96-35054f4e2171"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone/oxz_crucible_95fbeccd-695b-4f6e-ae96-35054f4e2171" } file = sled-storage/src/manager.rs:1431
5482025-01-11T01:17:50.599ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
5492025-01-11T01:17:50.623ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5502025-01-11T01:17:50.623ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5512025-01-11T01:17:50.623ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone/oxz_crucible_00711fcf-5ea5-4dbf-918c-e21236b2cac3"), full_name: "oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone/oxz_crucible_00711fcf-5ea5-4dbf-918c-e21236b2cac3" } file = sled-storage/src/manager.rs:1431
5522025-01-11T01:17:50.678ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5532025-01-11T01:17:50.678ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5542025-01-11T01:17:50.678ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone"), full_name: "oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone" } file = sled-storage/src/manager.rs:1431
5552025-01-11T01:17:50.707ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5562025-01-11T01:17:50.707ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5572025-01-11T01:17:50.707ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone"), full_name: "oxp_094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone" } file = sled-storage/src/manager.rs:1431
5582025-01-11T01:17:50.737ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5592025-01-11T01:17:50.737ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5602025-01-11T01:17:50.737ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/zone"), full_name: "oxp_a864efea-d3d9-4be0-84a5-f7e73206c783/crypt/zone" } file = sled-storage/src/manager.rs:1431
5612025-01-11T01:17:50.766ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5622025-01-11T01:17:50.766ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5632025-01-11T01:17:50.766ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_cockroachdb_1877f8e9-a377-4960-91f4-09dd1c711f3b"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_cockroachdb_1877f8e9-a377-4960-91f4-09dd1c711f3b" } file = sled-storage/src/manager.rs:1431
5642025-01-11T01:17:50.824ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5652025-01-11T01:17:50.824ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5662025-01-11T01:17:50.824ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crucible" } file = sled-storage/src/manager.rs:1431
5672025-01-11T01:17:50.870ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5682025-01-11T01:17:50.870ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5692025-01-11T01:17:50.870ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone/oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b"), full_name: "oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crypt/zone/oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b" } file = sled-storage/src/manager.rs:1431
5702025-01-11T01:17:50.928ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5712025-01-11T01:17:50.929ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5722025-01-11T01:17:50.929ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_a864efea-d3d9-4be0-84a5-f7e73206c783/crucible" } file = sled-storage/src/manager.rs:1431
5732025-01-11T01:17:50.975ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5742025-01-11T01:17:50.975ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5752025-01-11T01:17:50.975ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone/oxz_cockroachdb_dbbd09fb-c1a9-46dd-b9f4-782842395cbe"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone/oxz_cockroachdb_dbbd09fb-c1a9-46dd-b9f4-782842395cbe" } file = sled-storage/src/manager.rs:1431
5762025-01-11T01:17:51.038ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5772025-01-11T01:17:51.038ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5782025-01-11T01:17:51.038ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone/oxz_crucible_38bf453c-46b6-4fd8-b901-1a35d301feef"), full_name: "oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone/oxz_crucible_38bf453c-46b6-4fd8-b901-1a35d301feef" } file = sled-storage/src/manager.rs:1431
5792025-01-11T01:17:51.095ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5802025-01-11T01:17:51.095ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5812025-01-11T01:17:51.097ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_70afe259-ef7a-419a-97ed-388d46a1cb17/crucible" } file = sled-storage/src/manager.rs:1431
5822025-01-11T01:17:51.142ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5832025-01-11T01:17:51.143ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5842025-01-11T01:17:51.143ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/zone" } file = sled-storage/src/manager.rs:1431
5852025-01-11T01:17:51.172ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5862025-01-11T01:17:51.172ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5872025-01-11T01:17:51.172ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone/oxz_crucible_pantry_d70ae7d1-a48f-47f5-8d1a-20559576d4aa"), full_name: "oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/zone/oxz_crucible_pantry_d70ae7d1-a48f-47f5-8d1a-20559576d4aa" } file = sled-storage/src/manager.rs:1431
5882025-01-11T01:17:51.232ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5892025-01-11T01:17:51.232ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5902025-01-11T01:17:51.232ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/0a1d8462-3db8-4b03-acdf-9503499e711d/crypt/zone/oxz_crucible_4ee703ec-7859-4e24-b277-ed30a36988ee"), full_name: "oxp_0a1d8462-3db8-4b03-acdf-9503499e711d/crypt/zone/oxz_crucible_4ee703ec-7859-4e24-b277-ed30a36988ee" } file = sled-storage/src/manager.rs:1431
5912025-01-11T01:17:51.290ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5922025-01-11T01:17:51.290ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
5932025-01-11T01:17:51.290ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/zone"), full_name: "oxp_70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/zone" } file = sled-storage/src/manager.rs:1431
5942025-01-11T01:17:51.320ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5952025-01-11T01:17:51.320ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5962025-01-11T01:17:51.320ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone/oxz_crucible_fb142081-54df-4946-b831-3eb44893e9e0"), full_name: "oxp_094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone/oxz_crucible_fb142081-54df-4946-b831-3eb44893e9e0" } file = sled-storage/src/manager.rs:1431
5972025-01-11T01:17:51.377ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
5982025-01-11T01:17:51.377ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
5992025-01-11T01:17:51.377ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/zone/oxz_nexus_027e6e69-94cf-42dd-a83a-77f1fe49eb61"), full_name: "oxp_70afe259-ef7a-419a-97ed-388d46a1cb17/crypt/zone/oxz_nexus_027e6e69-94cf-42dd-a83a-77f1fe49eb61" } file = sled-storage/src/manager.rs:1431
6002025-01-11T01:17:51.437ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6012025-01-11T01:17:51.437ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6022025-01-11T01:17:51.437ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone/oxz_crucible_pantry_9107f0ba-d732-48cb-866e-8ec55c9ba80a"), full_name: "oxp_094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone/oxz_crucible_pantry_9107f0ba-d732-48cb-866e-8ec55c9ba80a" } file = sled-storage/src/manager.rs:1431
6032025-01-11T01:17:51.494ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6042025-01-11T01:17:51.494ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6052025-01-11T01:17:51.494ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8d41776e-a84a-44c7-a848-bc443cc63109/crypt/debug"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/debug" } file = sled-storage/src/manager.rs:1431
6062025-01-11T01:17:51.523ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6072025-01-11T01:17:51.523ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6082025-01-11T01:17:51.523ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_external_dns_cca5bf73-add8-4186-9f70-0867e2687634"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_external_dns_cca5bf73-add8-4186-9f70-0867e2687634" } file = sled-storage/src/manager.rs:1431
6092025-01-11T01:17:51.585ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6102025-01-11T01:17:51.585ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6112025-01-11T01:17:51.585ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_crucible_63191500-b9e6-45be-85a4-cdf6399d2f2a"), full_name: "oxp_8d41776e-a84a-44c7-a848-bc443cc63109/crypt/zone/oxz_crucible_63191500-b9e6-45be-85a4-cdf6399d2f2a" } file = sled-storage/src/manager.rs:1431
6122025-01-11T01:17:51.601ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
6132025-01-11T01:17:51.644ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6142025-01-11T01:17:51.644ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6152025-01-11T01:17:51.644ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_external_dns_f84184b7-030d-4a37-95a3-1d6ee7166492"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_external_dns_f84184b7-030d-4a37-95a3-1d6ee7166492" } file = sled-storage/src/manager.rs:1431
6162025-01-11T01:17:51.705ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6172025-01-11T01:17:51.705ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6182025-01-11T01:17:51.705ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1431
6192025-01-11T01:17:51.755ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6202025-01-11T01:17:51.755ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6212025-01-11T01:17:51.755ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_clickhouse_e5ee12e5-569a-47e9-b44f-6d568f6ce98c"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_clickhouse_e5ee12e5-569a-47e9-b44f-6d568f6ce98c" } file = sled-storage/src/manager.rs:1431
6222025-01-11T01:17:51.816ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6232025-01-11T01:17:51.816ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6242025-01-11T01:17:51.816ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/debug"), full_name: "oxp_3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b/crypt/debug" } file = sled-storage/src/manager.rs:1431
6252025-01-11T01:17:51.844ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6262025-01-11T01:17:51.844ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6272025-01-11T01:17:51.844ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_f5232c69-2469-4824-a4f8-901936b93f3d/crucible" } file = sled-storage/src/manager.rs:1431
6282025-01-11T01:17:51.894ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6292025-01-11T01:17:51.894ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6302025-01-11T01:17:51.894ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_crucible_pantry_edb48777-436b-4f70-bed2-e6896c7ded23"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_crucible_pantry_edb48777-436b-4f70-bed2-e6896c7ded23" } file = sled-storage/src/manager.rs:1431
6312025-01-11T01:17:51.953ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6322025-01-11T01:17:51.953ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6332025-01-11T01:17:51.953ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/clickhouse" } file = sled-storage/src/manager.rs:1431
6342025-01-11T01:17:52.004ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6352025-01-11T01:17:52.004ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6362025-01-11T01:17:52.004ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293"), full_name: "oxp_a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8/crypt/zone/oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293" } file = sled-storage/src/manager.rs:1431
6372025-01-11T01:17:52.063ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6382025-01-11T01:17:52.063ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6392025-01-11T01:17:52.063ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/0a1d8462-3db8-4b03-acdf-9503499e711d/crypt/zone"), full_name: "oxp_0a1d8462-3db8-4b03-acdf-9503499e711d/crypt/zone" } file = sled-storage/src/manager.rs:1431
6402025-01-11T01:17:52.092ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6412025-01-11T01:17:52.092ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6422025-01-11T01:17:52.092ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/0a1d8462-3db8-4b03-acdf-9503499e711d/crypt/debug"), full_name: "oxp_0a1d8462-3db8-4b03-acdf-9503499e711d/crypt/debug" } file = sled-storage/src/manager.rs:1431
6432025-01-11T01:17:52.123ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6442025-01-11T01:17:52.123ZINFOSledAgent (StorageManager): This dataset did not exist file = sled-storage/src/manager.rs:993 request = datasets_ensure
6452025-01-11T01:17:52.123ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone/oxz_nexus_21413c29-505f-4c82-9c8b-50571ddaefa5"), full_name: "oxp_094891fc-36ea-4f74-b7ba-91566beb4534/crypt/zone/oxz_nexus_21413c29-505f-4c82-9c8b-50571ddaefa5" } file = sled-storage/src/manager.rs:1431
6462025-01-11T01:17:52.186ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:1039 request = datasets_ensure
6472025-01-11T01:17:52.186ZINFOSledAgent (StorageManager): Old properties missing UUID file = sled-storage/src/manager.rs:998 request = datasets_ensure
6482025-01-11T01:17:52.186ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/debug"), full_name: "oxp_1d1a100e-9784-4f32-981f-1a3200e989b4/crypt/debug" } file = sled-storage/src/manager.rs:1431
6492025-01-11T01:17:52.219ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/.omicron-datasets.json.tmp file = common/src/ledger.rs:199 request = datasets_ensure
6502025-01-11T01:17:52.220ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/.omicron-datasets.json.tmp file = common/src/ledger.rs:199 request = datasets_ensure
6512025-01-11T01:17:52.220ZINFOSledAgent: datasets ensure: Updated storage file = sled-agent/src/sled_agent.rs:861 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
6522025-01-11T01:17:52.220ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 3184452 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51820 req_id = 879a08f8-4b5b-4f2e-8e6c-a40f81291539 response_code = 200 uri = /datasets
6532025-01-11T01:17:52.252ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 63a7090e-2b4b-4dc9-bed6-0a8e375f0293 (service), filesystem_pool: Some(ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0619e98e-96d1-41bb-8d2f-36679c84240f (service), filesystem_pool: Some(ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 69b23a24-f9c6-4e95-90fe-20e66106244e (service), filesystem_pool: Some(ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
6542025-01-11T01:17:52.252ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59100
6552025-01-11T01:17:52.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
6562025-01-11T01:17:52.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6572025-01-11T01:17:52.263ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
6582025-01-11T01:17:52.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6592025-01-11T01:17:52.273ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
6602025-01-11T01:17:52.273ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: InternalDns }, responder: ... }
6612025-01-11T01:17:52.281ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6622025-01-11T01:17:52.281ZINFOSledAgent (ServiceManager): No ledger in /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/omicron-zones.json file = common/src/ledger.rs:188
6632025-01-11T01:17:52.281ZINFOSledAgent (ServiceManager): No ledger in /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/omicron-zones.json file = common/src/ledger.rs:188
6642025-01-11T01:17:52.281ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6652025-01-11T01:17:52.325ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6662025-01-11T01:17:52.325ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6672025-01-11T01:17:52.325ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
6682025-01-11T01:17:52.376ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f file = illumos-utils/src/zone.rs:325
6692025-01-11T01:17:52.379ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e file = illumos-utils/src/zone.rs:325
6702025-01-11T01:17:52.383ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293 file = illumos-utils/src/zone.rs:325
6712025-01-11T01:17:52.396ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f file = illumos-utils/src/zone.rs:363
6722025-01-11T01:17:52.401ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e file = illumos-utils/src/zone.rs:363
6732025-01-11T01:17:52.404ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293 file = illumos-utils/src/zone.rs:363
6742025-01-11T01:17:52.602ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dfd15883-50bd-419a-82e1-b0ed2f66c5e6) file = sled-agent/src/probe_manager.rs:231 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
6752025-01-11T01:17:54.050ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6762025-01-11T01:17:54.051ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
6772025-01-11T01:17:54.282ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4467
6782025-01-11T01:17:54.282ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4494
6792025-01-11T01:17:54.312ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5300053050309105182 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
6802025-01-11T01:17:54.312ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9777168582998979993 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
6812025-01-11T01:17:54.321ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4507
6822025-01-11T01:17:54.321ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4353
6832025-01-11T01:17:54.371ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4395
6842025-01-11T01:17:54.371ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4474 rack_subnet = fd00:1122:3344::
6852025-01-11T01:17:54.390ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4485
6862025-01-11T01:17:54.390ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4527
6872025-01-11T01:17:54.457ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4559
6882025-01-11T01:17:54.457ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4565
6892025-01-11T01:17:54.533ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6902025-01-11T01:17:54.543ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4606
6912025-01-11T01:17:54.543ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
6922025-01-11T01:17:54.683ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 51.924638ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
6932025-01-11T01:17:54.736ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 136.807669ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
6942025-01-11T01:17:54.874ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:420
6952025-01-11T01:17:54.906ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:717
6962025-01-11T01:17:54.906ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:762
6972025-01-11T01:17:54.907ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:765 uptime = 0
6982025-01-11T01:17:54.907ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: Some(\n None,\n ),\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n tx_eq: None,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:444
6992025-01-11T01:17:54.943ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4145
7002025-01-11T01:17:54.992ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:607
7012025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
7022025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
7032025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
7042025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
7052025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
7062025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
7072025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
7082025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
7092025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
7102025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
7112025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
7122025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
7132025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
7142025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
7152025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
7162025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
7172025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
7182025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
7192025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
7202025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
7212025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
7222025-01-11T01:17:54.992ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
7232025-01-11T01:17:55.015ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
7242025-01-11T01:17:55.015ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7252025-01-11T01:17:55.015ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7262025-01-11T01:17:55.019ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
7272025-01-11T01:17:55.019ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7282025-01-11T01:17:55.024ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7292025-01-11T01:17:55.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 71.387042ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7302025-01-11T01:17:55.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 39.143072ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7312025-01-11T01:17:55.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 59.478572ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7322025-01-11T01:17:55.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 95.925284ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7332025-01-11T01:17:55.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 93.61745ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7342025-01-11T01:17:55.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 53.340202ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7352025-01-11T01:17:55.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 268.38658ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7362025-01-11T01:17:55.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 254.098503ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7372025-01-11T01:17:55.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 259.45305ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7382025-01-11T01:17:55.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 428.439545ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7392025-01-11T01:17:55.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 512.6165ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7402025-01-11T01:17:55.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 463.185535ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7412025-01-11T01:17:55.985ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
7422025-01-11T01:17:55.985ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
7432025-01-11T01:17:56.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 908.588386ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7442025-01-11T01:17:56.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 496.588706ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7452025-01-11T01:17:56.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 546.552527ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7462025-01-11T01:17:56.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 626.916612ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7472025-01-11T01:17:56.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 891.439697ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7482025-01-11T01:17:57.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 1.29816112s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7492025-01-11T01:17:57.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 760.355194ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7502025-01-11T01:17:57.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 1.265081531s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7512025-01-11T01:17:58.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 940.376177ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7522025-01-11T01:17:58.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 517.438896ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7532025-01-11T01:17:58.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 819.25562ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7542025-01-11T01:17:59.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 548.397307ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7552025-01-11T01:17:59.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 567.221787ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7562025-01-11T01:17:59.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293") failed: Property not found. retry in 1.457004548s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7572025-01-11T01:17:59.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 510.255534ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7582025-01-11T01:17:59.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e") failed: Property not found. retry in 1.147763055s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7592025-01-11T01:18:00.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f") failed: Property not found. retry in 558.186106ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7602025-01-11T01:18:01.051ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3336 zone = oxz_internal_dns_0619e98e-96d1-41bb-8d2f-36679c84240f
7612025-01-11T01:18:01.051ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3336 zone = oxz_internal_dns_69b23a24-f9c6-4e95-90fe-20e66106244e
7622025-01-11T01:18:01.051ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3336 zone = oxz_internal_dns_63a7090e-2b4b-4dc9-bed6-0a8e375f0293
7632025-01-11T01:18:01.051ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7642025-01-11T01:18:01.051ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7652025-01-11T01:18:01.051ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8798981 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59100 req_id = 612efa53-7657-4b0f-b0de-34d971172dde response_code = 204 uri = /omicron-zones
7662025-01-11T01:18:01.053ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:658
7672025-01-11T01:18:01.636ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 135.763792ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:678
7682025-01-11T01:18:01.792ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:658
7692025-01-11T01:18:01.839ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:658
7702025-01-11T01:18:01.887ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:694
7712025-01-11T01:18:01.887ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
7722025-01-11T01:18:01.920ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
7732025-01-11T01:18:01.920ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
7742025-01-11T01:18:01.920ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
7752025-01-11T01:18:01.950ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 63a7090e-2b4b-4dc9-bed6-0a8e375f0293 (service), filesystem_pool: Some(ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0619e98e-96d1-41bb-8d2f-36679c84240f (service), filesystem_pool: Some(ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 69b23a24-f9c6-4e95-90fe-20e66106244e (service), filesystem_pool: Some(ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 699de3c0-293d-43c3-8434-a57a6e89759b (service), filesystem_pool: Some(ZpoolName { id: f5232c69-2469-4824-a4f8-901936b93f3d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 38ad58e5-0e2c-44c0-be50-be45adea57c9, kind: Service { id: 699de3c0-293d-43c3-8434-a57a6e89759b }, name: Name("ntp-699de3c0-293d-43c3-8434-a57a6e89759b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 92])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:541 sled_address = [fd00:1122:3344:101::1]:12345
7762025-01-11T01:18:01.950ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47888
7772025-01-11T01:18:01.950ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
7782025-01-11T01:18:01.950ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: a3b0f374-f5e0-4dee-9a37-ffb49c3bc4e8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7792025-01-11T01:18:01.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
7802025-01-11T01:18:01.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 8d41776e-a84a-44c7-a848-bc443cc63109 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7812025-01-11T01:18:01.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:607
7822025-01-11T01:18:01.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1478 request = NewFilesystemRequest { dataset_id: None, dataset_name: DatasetName { pool_name: ZpoolName { id: 3f7bc1a3-3a0a-44ab-a7b2-f71b8c7f753b (zpool), kind: External }, kind: InternalDns }, responder: ... }
7832025-01-11T01:18:01.982ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
7842025-01-11T01:18:01.982ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/omicron-zones.json file = common/src/ledger.rs:177
7852025-01-11T01:18:01.982ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/omicron-zones.json file = common/src/ledger.rs:177
7862025-01-11T01:18:01.982ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
7872025-01-11T01:18:01.993ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
7882025-01-11T01:18:02.025ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
7892025-01-11T01:18:02.027ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
7902025-01-11T01:18:02.027ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
7912025-01-11T01:18:02.057ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:447 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 229, 92]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
7922025-01-11T01:18:02.057ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1275 zone_type = boundary_ntp
7932025-01-11T01:18:02.057ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:85
7942025-01-11T01:18:02.057ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:92
7952025-01-11T01:18:02.058ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:120
7962025-01-11T01:18:02.058ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:607
7972025-01-11T01:18:02.082ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b file = illumos-utils/src/zone.rs:325
7982025-01-11T01:18:02.101ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b file = illumos-utils/src/zone.rs:363
7992025-01-11T01:18:02.725ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8002025-01-11T01:18:02.725ZINFOSledAgent (ServiceManager): Booting oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8012025-01-11T01:18:02.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 59.590134ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8022025-01-11T01:18:02.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 132.202838ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8032025-01-11T01:18:03.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 203.017483ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8042025-01-11T01:18:03.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 313.307ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8052025-01-11T01:18:03.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 604.034196ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8062025-01-11T01:18:04.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 502.065767ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8072025-01-11T01:18:04.315ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5300053050309105182 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8082025-01-11T01:18:04.315ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9777168582998979993 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8092025-01-11T01:18:04.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 681.403709ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8102025-01-11T01:18:05.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 547.114092ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8112025-01-11T01:18:05.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 1.495711008s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8122025-01-11T01:18:05.987ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8132025-01-11T01:18:05.987ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8142025-01-11T01:18:07.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b") failed: Property not found. retry in 1.206429017s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8152025-01-11T01:18:08.713ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3336 zone = oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b
8162025-01-11T01:18:08.713ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2026aef5-ecc1-428c-aac9-f90b84f99582/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8172025-01-11T01:18:08.713ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/9404c118-e76c-41a3-b09c-850793e381f3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8182025-01-11T01:18:08.714ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 6763577 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47888 req_id = d1deea3f-2974-4316-ac8e-94c4e6d661a1 response_code = 204 uri = /omicron-zones
8192025-01-11T01:18:08.716ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:735
8202025-01-11T01:18:08.747ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8212025-01-11T01:18:08.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33166
8222025-01-11T01:18:08.756ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3926
8232025-01-11T01:18:08.756ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 8560 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33166 req_id = a1f20e27-3c0b-46e0-94b1-203b95a88536 response_code = 500 uri = /timesync
8242025-01-11T01:18:08.758ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8252025-01-11T01:18:08.856ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8262025-01-11T01:18:08.856ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53385
8272025-01-11T01:18:10.777ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16432527369878710882 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8282025-01-11T01:18:10.885ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15596257807682548297 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8292025-01-11T01:18:11.053ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8597029270196437948 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8302025-01-11T01:18:12.865ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3926
8312025-01-11T01:18:12.865ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 4008777 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53385 req_id = d382c41a-036a-42cd-b609-bdbd18f874f6 response_code = 500 uri = /timesync
8322025-01-11T01:18:12.868ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8332025-01-11T01:18:13.021ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8342025-01-11T01:18:13.021ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36732
8352025-01-11T01:18:13.030ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3926
8362025-01-11T01:18:13.030ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 8500 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36732 req_id = 6f6e8b95-8fea-4abf-bd9c-f303919e9aef response_code = 500 uri = /timesync
8372025-01-11T01:18:13.032ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8382025-01-11T01:18:13.264ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8392025-01-11T01:18:13.265ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44152
8402025-01-11T01:18:13.273ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_699de3c0-293d-43c3-8434-a57a6e89759b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3926
8412025-01-11T01:18:13.273ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 8551 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44152 req_id = e4eceaa0-a740-4b48-ab86-4b40dc803ab7 response_code = 500 uri = /timesync
8422025-01-11T01:18:13.276ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8432025-01-11T01:18:13.525ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8442025-01-11T01:18:13.525ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48775
8452025-01-11T01:18:13.534ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8384 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48775 req_id = fc2e94be-ed79-4048-b773-dffcff10c2ca response_code = 200 uri = /timesync
8462025-01-11T01:18:13.537ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8472025-01-11T01:18:13.537ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8482025-01-11T01:18:13.691ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8492025-01-11T01:18:13.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64373
8502025-01-11T01:18:13.699ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7350 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64373 req_id = f22b17d6-af87-4414-b23b-355c15c57bd1 response_code = 200 uri = /timesync
8512025-01-11T01:18:13.700ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8522025-01-11T01:18:13.700ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8532025-01-11T01:18:14.098ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8542025-01-11T01:18:14.098ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56540
8552025-01-11T01:18:14.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 10406 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56540 req_id = 3aec9976-da6d-40f8-9a13-2dd3957f5ef8 response_code = 200 uri = /timesync
8562025-01-11T01:18:14.112ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8572025-01-11T01:18:14.112ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8582025-01-11T01:18:14.318ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5300053050309105182 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8592025-01-11T01:18:14.318ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9777168582998979993 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8602025-01-11T01:18:14.396ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8612025-01-11T01:18:14.396ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54187
8622025-01-11T01:18:14.404ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7811 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54187 req_id = aaa65c03-e6b0-4cc1-a49a-bf06d4333e76 response_code = 200 uri = /timesync
8632025-01-11T01:18:14.406ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8642025-01-11T01:18:14.406ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8652025-01-11T01:18:14.753ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8662025-01-11T01:18:14.753ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37660
8672025-01-11T01:18:14.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8751 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37660 req_id = 15555fc5-4759-4a73-96a8-c809a684bffa response_code = 200 uri = /timesync
8682025-01-11T01:18:14.763ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8692025-01-11T01:18:14.763ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8702025-01-11T01:18:15.437ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8712025-01-11T01:18:15.437ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35325
8722025-01-11T01:18:15.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9332 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35325 req_id = b63c6651-aee5-4688-aa09-3f0121b4fa3c response_code = 200 uri = /timesync
8732025-01-11T01:18:15.450ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8742025-01-11T01:18:15.450ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8752025-01-11T01:18:15.998ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8762025-01-11T01:18:15.998ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40891
8772025-01-11T01:18:16.004ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8782025-01-11T01:18:16.004ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8792025-01-11T01:18:16.008ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9537 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40891 req_id = 83cb7f40-1b28-43e4-82af-ace559676e58 response_code = 200 uri = /timesync
8802025-01-11T01:18:16.011ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8812025-01-11T01:18:16.011ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8822025-01-11T01:18:16.637ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8832025-01-11T01:18:16.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60325
8842025-01-11T01:18:16.645ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8066 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60325 req_id = 9ec0b2ce-2e29-42f8-90c1-efe083ec9ff7 response_code = 200 uri = /timesync
8852025-01-11T01:18:16.647ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8862025-01-11T01:18:16.647ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8872025-01-11T01:18:17.732ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8882025-01-11T01:18:17.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40435
8892025-01-11T01:18:17.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8826 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40435 req_id = 9b410e83-54a0-4450-ae32-43704ea7a4ee response_code = 200 uri = /timesync
8902025-01-11T01:18:17.743ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8912025-01-11T01:18:17.743ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8922025-01-11T01:18:18.716ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2768020810680673949 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8932025-01-11T01:18:18.717ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12050495470882410828 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
8942025-01-11T01:18:19.028ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
8952025-01-11T01:18:19.028ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35497
8962025-01-11T01:18:19.037ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8554 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35497 req_id = fbd06fbb-5379-4f60-8102-8dfdf85f7468 response_code = 200 uri = /timesync
8972025-01-11T01:18:19.041ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
8982025-01-11T01:18:19.041ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
8992025-01-11T01:18:19.384ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 256.481874ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9002025-01-11T01:18:20.246ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9012025-01-11T01:18:20.246ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52019
9022025-01-11T01:18:20.255ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9003 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52019 req_id = e42aa776-86b2-46a4-a9dd-dd27f4b86b97 response_code = 200 uri = /timesync
9032025-01-11T01:18:20.257ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9042025-01-11T01:18:20.257ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9052025-01-11T01:18:20.778ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16432527369878710882 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9062025-01-11T01:18:20.886ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15596257807682548297 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9072025-01-11T01:18:21.055ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8597029270196437948 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9082025-01-11T01:18:21.467ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9092025-01-11T01:18:21.467ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57308
9102025-01-11T01:18:21.476ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8934 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57308 req_id = f359405d-5236-4df1-8098-feb151379d56 response_code = 200 uri = /timesync
9112025-01-11T01:18:21.478ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9122025-01-11T01:18:21.478ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9132025-01-11T01:18:22.628ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9142025-01-11T01:18:22.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46161
9152025-01-11T01:18:22.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8393 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46161 req_id = 12d7d0df-2141-4b14-8aae-6676e5e55b78 response_code = 200 uri = /timesync
9162025-01-11T01:18:22.639ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9172025-01-11T01:18:22.639ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9182025-01-11T01:18:24.250ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9192025-01-11T01:18:24.250ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42701
9202025-01-11T01:18:24.258ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8382 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42701 req_id = 5e0ab79c-8788-4357-88b9-42263ff26974 response_code = 200 uri = /timesync
9212025-01-11T01:18:24.263ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9222025-01-11T01:18:24.263ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9232025-01-11T01:18:24.320ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5300053050309105182 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9242025-01-11T01:18:24.320ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9777168582998979993 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9252025-01-11T01:18:24.890ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:44237
9262025-01-11T01:18:24.890ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 76 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:44237 req_id = 77370905-3a0d-460d-8151-ed025c9aa8be response_code = 200 uri = /baseboard
9272025-01-11T01:18:26.006ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9282025-01-11T01:18:26.006ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9292025-01-11T01:18:26.135ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9302025-01-11T01:18:26.135ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36791
9312025-01-11T01:18:26.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9450 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36791 req_id = 8f438514-e62e-476c-a299-8908aeb95f54 response_code = 200 uri = /timesync
9322025-01-11T01:18:26.148ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9332025-01-11T01:18:26.148ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9342025-01-11T01:18:28.719ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2768020810680673949 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9352025-01-11T01:18:28.719ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12050495470882410828 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9362025-01-11T01:18:29.647ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9372025-01-11T01:18:29.648ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48388
9382025-01-11T01:18:29.656ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8509 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48388 req_id = 3b1089d9-99e0-4995-a06c-e94ca4e18c72 response_code = 200 uri = /timesync
9392025-01-11T01:18:29.660ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9402025-01-11T01:18:29.660ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9412025-01-11T01:18:30.780ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16432527369878710882 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9422025-01-11T01:18:30.887ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15596257807682548297 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9432025-01-11T01:18:31.057ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8597029270196437948 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9442025-01-11T01:18:32.765ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9452025-01-11T01:18:32.765ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62808
9462025-01-11T01:18:32.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9235 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62808 req_id = 8e7aab84-644f-47e9-8f94-71ebedcbd874 response_code = 200 uri = /timesync
9472025-01-11T01:18:32.777ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9482025-01-11T01:18:32.777ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9492025-01-11T01:18:34.323ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5300053050309105182 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9502025-01-11T01:18:34.323ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9777168582998979993 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9512025-01-11T01:18:34.675ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 564.763778ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9522025-01-11T01:18:36.008ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9532025-01-11T01:18:36.008ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9542025-01-11T01:18:36.139ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9552025-01-11T01:18:36.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39014
9562025-01-11T01:18:36.148ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8952 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39014 req_id = 4b7898b5-ee28-4851-86f7-c3b2ffbe565e response_code = 200 uri = /timesync
9572025-01-11T01:18:36.153ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9582025-01-11T01:18:36.153ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9592025-01-11T01:18:38.721ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2768020810680673949 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9602025-01-11T01:18:38.721ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12050495470882410828 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9612025-01-11T01:18:39.848ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9622025-01-11T01:18:39.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46480
9632025-01-11T01:18:39.856ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8516 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46480 req_id = 0c894136-cf06-44c0-aa6a-f8fd8aed6ebd response_code = 200 uri = /timesync
9642025-01-11T01:18:39.859ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9652025-01-11T01:18:39.859ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9662025-01-11T01:18:40.783ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16432527369878710882 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9672025-01-11T01:18:40.888ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15596257807682548297 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9682025-01-11T01:18:41.057ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8597029270196437948 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9692025-01-11T01:18:43.034ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9702025-01-11T01:18:43.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33372
9712025-01-11T01:18:43.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9088 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33372 req_id = 899945ca-ab81-4ae9-a20d-1294d575cc51 response_code = 200 uri = /timesync
9722025-01-11T01:18:43.045ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9732025-01-11T01:18:43.045ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9742025-01-11T01:18:44.324ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 5300053050309105182 reason = Duration(60s) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9752025-01-11T01:18:44.325ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 9777168582998979993 reason = Duration(60s) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9762025-01-11T01:18:46.011ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9772025-01-11T01:18:46.011ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9782025-01-11T01:18:48.723ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2768020810680673949 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9792025-01-11T01:18:48.723ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12050495470882410828 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9802025-01-11T01:18:48.993ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9812025-01-11T01:18:48.993ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58638
9822025-01-11T01:18:49.001ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8536 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58638 req_id = 6efb64b0-15b3-4751-8787-7b4fbcf8814f response_code = 200 uri = /timesync
9832025-01-11T01:18:49.004ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9842025-01-11T01:18:49.004ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9852025-01-11T01:18:50.277ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.069473735s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9862025-01-11T01:18:50.784ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16432527369878710882 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9872025-01-11T01:18:50.889ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15596257807682548297 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9882025-01-11T01:18:51.059ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8597029270196437948 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9892025-01-11T01:18:54.925ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:36063
9902025-01-11T01:18:54.925ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 46 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:36063 req_id = 12f4972d-9a60-4465-8af0-df3eaad33438 response_code = 200 uri = /baseboard
9912025-01-11T01:18:56.013ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9922025-01-11T01:18:56.013ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9932025-01-11T01:18:57.395ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
9942025-01-11T01:18:57.395ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50224
9952025-01-11T01:18:57.404ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8905 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50224 req_id = 065caaa6-1a5a-48de-bece-481b60f38b5e response_code = 200 uri = /timesync
9962025-01-11T01:18:57.407ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
9972025-01-11T01:18:57.407ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
9982025-01-11T01:18:58.725ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2768020810680673949 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
9992025-01-11T01:18:58.725ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12050495470882410828 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10002025-01-11T01:19:00.787ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 16432527369878710882 reason = Duration(60s) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10012025-01-11T01:19:00.890ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 15596257807682548297 reason = Duration(60s) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10022025-01-11T01:19:01.060ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 8597029270196437948 reason = Duration(60s) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10032025-01-11T01:19:06.015ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10042025-01-11T01:19:06.015ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10052025-01-11T01:19:06.382ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.79050156s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10062025-01-11T01:19:08.726ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 2768020810680673949 reason = Duration(60s) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10072025-01-11T01:19:08.726ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 12050495470882410828 reason = Duration(60s) sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10082025-01-11T01:19:09.901ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10092025-01-11T01:19:09.901ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63317
10102025-01-11T01:19:09.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9170 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63317 req_id = f90653bc-62be-4f3b-9eaa-eaa4c1037509 response_code = 200 uri = /timesync
10112025-01-11T01:19:09.913ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10122025-01-11T01:19:09.913ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10132025-01-11T01:19:16.016ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10142025-01-11T01:19:16.016ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10152025-01-11T01:19:24.207ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 5.74934884s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10162025-01-11T01:19:24.959ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:59415
10172025-01-11T01:19:24.959ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 31 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:59415 req_id = b619a42c-2704-42a4-aa5b-fc4e98d4e3f1 response_code = 200 uri = /baseboard
10182025-01-11T01:19:25.762ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10192025-01-11T01:19:25.762ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64628
10202025-01-11T01:19:25.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8908 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64628 req_id = 4ce76e3b-8cc4-4b5f-a45a-7718c3162d5d response_code = 200 uri = /timesync
10212025-01-11T01:19:25.774ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10222025-01-11T01:19:25.774ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10232025-01-11T01:19:26.018ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10242025-01-11T01:19:26.018ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10252025-01-11T01:19:36.020ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10262025-01-11T01:19:36.021ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10272025-01-11T01:19:44.990ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.613578865s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10282025-01-11T01:19:46.009ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10292025-01-11T01:19:46.009ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57175
10302025-01-11T01:19:46.017ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8523 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57175 req_id = 8f45fa6d-d3aa-448d-a7d3-3a958a2378fa response_code = 200 uri = /timesync
10312025-01-11T01:19:46.020ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10322025-01-11T01:19:46.020ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10332025-01-11T01:19:46.023ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10342025-01-11T01:19:46.023ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10352025-01-11T01:19:54.993ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:45490
10362025-01-11T01:19:54.994ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 44 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:45490 req_id = 6cff8169-46a7-4ca2-9607-00e49c08a26d response_code = 200 uri = /baseboard
10372025-01-11T01:19:55.935ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10382025-01-11T01:19:55.935ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63208
10392025-01-11T01:19:55.945ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9757 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63208 req_id = 65f568b2-0f7f-4a22-9ab7-ef275cfd0924 response_code = 200 uri = /timesync
10402025-01-11T01:19:55.948ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10412025-01-11T01:19:55.948ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10422025-01-11T01:19:56.025ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10432025-01-11T01:19:56.025ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10442025-01-11T01:20:04.641ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 16.790021411s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10452025-01-11T01:20:06.026ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10462025-01-11T01:20:06.026ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10472025-01-11T01:20:16.029ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10482025-01-11T01:20:16.029ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10492025-01-11T01:20:17.283ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10502025-01-11T01:20:17.283ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59985
10512025-01-11T01:20:17.292ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8780 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59985 req_id = 05da00d3-ac87-463c-82a2-4d1be77d2206 response_code = 200 uri = /timesync
10522025-01-11T01:20:17.296ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10532025-01-11T01:20:17.296ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10542025-01-11T01:20:25.026ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:41293
10552025-01-11T01:20:25.026ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 35 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:41293 req_id = 4a62192d-8312-46df-b761-161c2ce262d9 response_code = 200 uri = /baseboard
10562025-01-11T01:20:26.031ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10572025-01-11T01:20:26.031ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10582025-01-11T01:20:34.157ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10592025-01-11T01:20:34.157ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43360
10602025-01-11T01:20:34.167ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9467 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43360 req_id = 43411580-f838-4ceb-b3c3-fc0b10f8b4ec response_code = 200 uri = /timesync
10612025-01-11T01:20:34.172ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10622025-01-11T01:20:34.172ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10632025-01-11T01:20:36.033ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10642025-01-11T01:20:36.033ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10652025-01-11T01:20:36.466ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 34.045936274s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10662025-01-11T01:20:46.036ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10672025-01-11T01:20:46.036ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10682025-01-11T01:20:55.059ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:50919
10692025-01-11T01:20:55.059ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 60 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:50919 req_id = 0efd8e27-3a63-4922-8e54-2a2283be41de response_code = 200 uri = /baseboard
10702025-01-11T01:20:56.037ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10712025-01-11T01:20:56.037ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10722025-01-11T01:20:56.785ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10732025-01-11T01:20:56.785ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37765
10742025-01-11T01:20:56.794ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8521 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37765 req_id = f13791f7-532d-4ef7-9259-987ff50ecf51 response_code = 200 uri = /timesync
10752025-01-11T01:20:56.799ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10762025-01-11T01:20:56.799ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10772025-01-11T01:21:06.039ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10782025-01-11T01:21:06.039ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10792025-01-11T01:21:16.041ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10802025-01-11T01:21:16.041ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10812025-01-11T01:21:25.092ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:39422
10822025-01-11T01:21:25.092ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 45 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:39422 req_id = 545696f8-2880-4dd7-9edc-fe27ca6f2640 response_code = 200 uri = /baseboard
10832025-01-11T01:21:25.547ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 56.626711824s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10842025-01-11T01:21:26.044ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10852025-01-11T01:21:26.044ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10862025-01-11T01:21:36.046ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10872025-01-11T01:21:36.046ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10882025-01-11T01:21:43.328ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
10892025-01-11T01:21:43.328ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37336
10902025-01-11T01:21:43.337ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9064 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37336 req_id = ff2a787e-dfb4-4aa7-ab3b-dc3b17ef454a response_code = 200 uri = /timesync
10912025-01-11T01:21:43.342ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
10922025-01-11T01:21:43.342ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
10932025-01-11T01:21:46.049ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10942025-01-11T01:21:46.049ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10952025-01-11T01:21:55.124ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:41111
10962025-01-11T01:21:55.124ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 36 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:41111 req_id = a373afa3-5eaa-4d6d-866b-d6c2f1fa86b4 response_code = 200 uri = /baseboard
10972025-01-11T01:21:56.050ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10982025-01-11T01:21:56.050ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
10992025-01-11T01:22:06.053ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11002025-01-11T01:22:06.053ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11012025-01-11T01:22:16.054ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11022025-01-11T01:22:16.054ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11032025-01-11T01:22:25.158ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:59060
11042025-01-11T01:22:25.158ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 39 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:59060 req_id = c7190b08-424f-4cc3-b65c-0c3c5c939307 response_code = 200 uri = /baseboard
11052025-01-11T01:22:26.057ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11062025-01-11T01:22:26.057ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11072025-01-11T01:22:26.427ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
11082025-01-11T01:22:26.428ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39115
11092025-01-11T01:22:26.437ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9245 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39115 req_id = 829fbccd-2345-4c13-b16f-88b0ff552bd0 response_code = 200 uri = /timesync
11102025-01-11T01:22:26.440ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
11112025-01-11T01:22:26.440ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
11122025-01-11T01:22:36.059ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11132025-01-11T01:22:36.059ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11142025-01-11T01:22:37.209ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 94.939681943s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11152025-01-11T01:22:46.061ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11162025-01-11T01:22:46.061ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11172025-01-11T01:22:49.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
11182025-01-11T01:22:55.192ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:54968
11192025-01-11T01:22:55.192ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 46 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:54968 req_id = 58d9c6a1-174c-4f19-8623-5b082ac86b57 response_code = 200 uri = /baseboard
11202025-01-11T01:22:56.062ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11212025-01-11T01:22:56.062ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11222025-01-11T01:22:59.174ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
11232025-01-11T01:22:59.174ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51029
11242025-01-11T01:22:59.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51029 req_id = d94750c0-cd9e-4cd3-992d-5e5da67a3f23 response_code = 200 uri = /timesync
11252025-01-11T01:22:59.188ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
11262025-01-11T01:22:59.188ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
11272025-01-11T01:23:06.065ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11282025-01-11T01:23:06.065ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11292025-01-11T01:23:16.067ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11302025-01-11T01:23:16.067ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11312025-01-11T01:23:25.223ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:33875
11322025-01-11T01:23:25.223ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 46 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:33875 req_id = 95abd020-28c0-400f-9af4-2f3f94ff636e response_code = 200 uri = /baseboard
11332025-01-11T01:23:26.069ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11342025-01-11T01:23:26.069ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11352025-01-11T01:23:35.215ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
11362025-01-11T01:23:35.215ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49147
11372025-01-11T01:23:35.224ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8489 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49147 req_id = 8e5d33ae-3e27-4233-a4eb-921da436c045 response_code = 200 uri = /timesync
11382025-01-11T01:23:35.226ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
11392025-01-11T01:23:35.226ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
11402025-01-11T01:23:36.071ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11412025-01-11T01:23:36.071ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11422025-01-11T01:23:46.073ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11432025-01-11T01:23:46.074ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11442025-01-11T01:23:55.258ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:61219
11452025-01-11T01:23:55.258ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 52 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:61219 req_id = e774fa2d-1a31-4c10-9606-a962f5fce045 response_code = 200 uri = /baseboard
11462025-01-11T01:23:56.075ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11472025-01-11T01:23:56.075ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11482025-01-11T01:24:06.078ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11492025-01-11T01:24:06.078ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11502025-01-11T01:24:16.080ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11512025-01-11T01:24:16.080ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11522025-01-11T01:24:19.112ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
11532025-01-11T01:24:19.112ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61657
11542025-01-11T01:24:19.121ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8714 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61657 req_id = 079a6cf1-7a11-471e-adcb-039bba2b27de response_code = 200 uri = /timesync
11552025-01-11T01:24:19.124ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
11562025-01-11T01:24:19.124ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
11572025-01-11T01:24:25.290ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:35953
11582025-01-11T01:24:25.291ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 46 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:35953 req_id = 18ad84be-077f-4af2-b0a3-a43a9abf05bb response_code = 200 uri = /baseboard
11592025-01-11T01:24:26.081ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11602025-01-11T01:24:26.082ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11612025-01-11T01:24:27.186ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 223.236929212s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11622025-01-11T01:24:36.084ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11632025-01-11T01:24:36.084ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11642025-01-11T01:24:46.086ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11652025-01-11T01:24:46.086ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11662025-01-11T01:24:55.323ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:44710
11672025-01-11T01:24:55.323ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 36 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:44710 req_id = 0f4b7003-88ce-4810-924d-d9f42af27e42 response_code = 200 uri = /baseboard
11682025-01-11T01:24:56.088ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11692025-01-11T01:24:56.088ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11702025-01-11T01:25:06.090ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11712025-01-11T01:25:06.090ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11722025-01-11T01:25:16.092ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11732025-01-11T01:25:16.092ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11742025-01-11T01:25:25.356ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:44639
11752025-01-11T01:25:25.356ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 36 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:44639 req_id = 34dba5d5-22d7-4c36-8eb1-9048defb3e77 response_code = 200 uri = /baseboard
11762025-01-11T01:25:26.093ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11772025-01-11T01:25:26.093ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11782025-01-11T01:25:27.217ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:715
11792025-01-11T01:25:27.217ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42727
11802025-01-11T01:25:27.227ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9066 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42727 req_id = fea5d7c6-93fb-44cf-a331-eba22b06d444 response_code = 200 uri = /timesync
11812025-01-11T01:25:27.229ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:743
11822025-01-11T01:25:27.229ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:765
11832025-01-11T01:25:36.095ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11842025-01-11T01:25:36.096ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11852025-01-11T01:25:46.098ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11862025-01-11T01:25:46.098ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11872025-01-11T01:25:55.389ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:54487
11882025-01-11T01:25:55.389ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 47 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:54487 req_id = 1e1ed818-a6ff-4cc9-a87f-079b81084be3 response_code = 200 uri = /baseboard
11892025-01-11T01:25:56.100ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11902025-01-11T01:25:56.100ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11912025-01-11T01:26:06.103ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11922025-01-11T01:26:06.103ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11932025-01-11T01:26:16.105ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11942025-01-11T01:26:16.105ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11952025-01-11T01:26:25.423ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:45774
11962025-01-11T01:26:25.423ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 39 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:45774 req_id = 03043d64-1010-4979-bb9d-4dfd8e483ae0 response_code = 200 uri = /baseboard
11972025-01-11T01:26:26.107ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11982025-01-11T01:26:26.107ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
11992025-01-11T01:26:36.108ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
12002025-01-11T01:26:36.108ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
12012025-01-11T01:26:46.110ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
12022025-01-11T01:26:46.110ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
12032025-01-11T01:26:55.456ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:40135
12042025-01-11T01:26:55.456ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 37 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:40135 req_id = f7078848-fb9f-43c0-9b07-3b9570819e45 response_code = 200 uri = /baseboard
12052025-01-11T01:26:56.111ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4153943977543823991 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6
12062025-01-11T01:26:56.112ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2612706593396418549 sled_id = dfd15883-50bd-419a-82e1-b0ed2f66c5e6