Max level shown:
1 [ Nov 21 23:09:58 Enabled. ]
2 [ Nov 21 23:09:58 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Nov 21 23:09:58 Method "start" exited with status 0. ]
4 [ Nov 21 23:09:58 Rereading configuration. ]
5 [ Nov 21 23:09:58 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-11-21T23:09:58.825ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-11-21T23:09:58.825ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-11-21T23:09:58.877ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-11-21T23:09:58.877ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-11-21T23:09:58.993ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-11-21T23:09:59.003ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-11-21T23:09:59.011ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-11-21T23:09:59.608ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:200::, width: 64 }
152024-11-21T23:09:59.743ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-11-21T23:09:59.743ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-11-21T23:09:59.743ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-11-21T23:09:59.743ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-11-21T23:09:59.743ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-11-21T23:09:59.743ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-11-21T23:09:59.746ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-11-21T23:09:59.746ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-11-21T23:09:59.746ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-11-21T23:09:59.746ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-11-21T23:09:59.746ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-11-21T23:09:59.746ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-11-21T23:09:59.746ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
282024-11-21T23:09:59.746ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-11-21T23:09:59.746ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-11-21T23:09:59.758ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-11-21T23:09:59.758ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool)
322024-11-21T23:09:59.858ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal }
332024-11-21T23:10:00.339ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:603
342024-11-21T23:10:00.339ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-11-21T23:10:00.339ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-11-21T23:10:00.339ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-11-21T23:10:00.339ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
382024-11-21T23:10:00.339ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-11-21T23:10:00.339ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-11-21T23:10:00.348ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412024-11-21T23:10:00.348ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool)
422024-11-21T23:10:00.436ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal }
432024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
472024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
482024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
522024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
582024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
602024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
662024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
722024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
752024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
762024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
772024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
782024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
812024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
822024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
832024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
872024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-11-21T23:10:00.922ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
932024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
942024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:603
952024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
962024-11-21T23:10:00.922ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
972024-11-21T23:10:00.922ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-11-21T23:10:00.922ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-11-21T23:10:00.922ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-11-21T23:10:00.922ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-11-21T23:10:00.922ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
1032024-11-21T23:10:00.922ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-11-21T23:10:00.922ZINFOSledAgent (bootstore): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:189 peer_id = pc-sock-i86pc
1052024-11-21T23:10:00.922ZINFOSledAgent (bootstore): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:189 peer_id = pc-sock-i86pc
1062024-11-21T23:10:00.922ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-11-21T23:10:00.922ZINFOSledAgent (bootstore): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/cluster/bootstore-network-config.json file = common/src/ledger.rs:189 peer_id = pc-sock-i86pc
1082024-11-21T23:10:00.922ZINFOSledAgent (bootstore): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/cluster/bootstore-network-config.json file = common/src/ledger.rs:189 peer_id = pc-sock-i86pc
1092024-11-21T23:10:00.922ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-11-21T23:10:00.922ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-11-21T23:10:00.922ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-11-21T23:10:00.933ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:743
1132024-11-21T23:10:00.933ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1388, tv_nsec: 805540073 } time_to_next_cleanup = 599.999917039s
1142024-11-21T23:10:00.933ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:4005
1152024-11-21T23:10:00.933ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
1162024-11-21T23:10:00.933ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/sled-agent-request.json file = common/src/ledger.rs:189
1172024-11-21T23:10:00.936ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/sled-agent-request.json file = common/src/ledger.rs:189
1182024-11-21T23:10:00.936ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fdb0:200::1]:80
1192024-11-21T23:10:00.938ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:200::1]:12346
1202024-11-21T23:10:00.938ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: Some( None, ), bgp_peers: [], autoneg: false, lldp: None, tx_eq: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1200
1212024-11-21T23:10:00.938ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1222024-11-21T23:10:00.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
1232024-11-21T23:10:00.938ZINFOSledAgent (RSS): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/rss-plan-completed.marker file = common/src/ledger.rs:189
1242024-11-21T23:10:00.938ZINFOSledAgent (RSS): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/rss-plan-completed.marker file = common/src/ledger.rs:189
1252024-11-21T23:10:00.938ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1263
1262024-11-21T23:10:00.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
1272024-11-21T23:10:00.938ZINFOSledAgent (RSS): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/rss-sled-plan.json file = common/src/ledger.rs:189
1282024-11-21T23:10:00.938ZINFOSledAgent (RSS): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/rss-sled-plan.json file = common/src/ledger.rs:189
1292024-11-21T23:10:00.938ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1305
1302024-11-21T23:10:00.938ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-11-21T23:10:00.938ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:200::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-11-21T23:10:00.939ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
1332024-11-21T23:10:00.939ZINFOSledAgent (RSS): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:200
1342024-11-21T23:10:00.939ZINFOSledAgent (RSS): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/rss-sled-plan.json"
1352024-11-21T23:10:00.939ZINFOSledAgent (RSS): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:200
1362024-11-21T23:10:00.939ZINFOSledAgent (RSS): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/rss-sled-plan.json"
1372024-11-21T23:10:00.939ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: c1b8264e-5605-46dd-a3d8-c05674dfacdb, sleds: { [fdb0:200::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: df02b17d-a4bd-4ac6-9447-5abe2628a534 (sled), rack_id: c1b8264e-5605-46dd-a3d8-c05674dfacdb, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: Some( None, ), bgp_peers: [], autoneg: false, lldp: None, tx_eq: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1382024-11-21T23:10:00.939ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1339
1392024-11-21T23:10:00.939ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1402024-11-21T23:10:00.939ZINFOSledAgent (bootstore): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:200 peer_id = pc-sock-i86pc
1412024-11-21T23:10:00.939ZINFOSledAgent (bootstore): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/cluster/bootstore-network-config.json" peer_id = pc-sock-i86pc
1422024-11-21T23:10:00.939ZINFOSledAgent (bootstore): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:200 peer_id = pc-sock-i86pc
1432024-11-21T23:10:00.939ZINFOSledAgent (bootstore): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/cluster/bootstore-network-config.json" peer_id = pc-sock-i86pc
1442024-11-21T23:10:00.939ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1452024-11-21T23:10:00.939ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: df02b17d-a4bd-4ac6-9447-5abe2628a534 (sled), rack_id: c1b8264e-5605-46dd-a3d8-c05674dfacdb, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:200::1]:12346
1462024-11-21T23:10:00.940ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4302
1472024-11-21T23:10:00.940ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4702
1482024-11-21T23:10:00.944ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:200::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1492024-11-21T23:10:00.950ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:200::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1502024-11-21T23:10:00.950ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:200::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1512024-11-21T23:10:00.954ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1522024-11-21T23:10:00.954ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1532024-11-21T23:10:00.955ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:200::1]:46004
1542024-11-21T23:10:00.955ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: df02b17d-a4bd-4ac6-9447-5abe2628a534 (sled), rack_id: c1b8264e-5605-46dd-a3d8-c05674dfacdb, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1552024-11-21T23:10:00.955ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1562024-11-21T23:10:00.955ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1572024-11-21T23:10:00.955ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1582024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:603
1592024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:809
1602024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-physical-disks.json file = common/src/ledger.rs:189 request = load_disks_ledger
1612024-11-21T23:10:00.955ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1622024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-physical-disks.json file = common/src/ledger.rs:189 request = load_disks_ledger
1632024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:798
1642024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:826
1652024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:592
1662024-11-21T23:10:00.955ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1672024-11-21T23:10:00.955ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:697 result = DisksManagementResult { status: [] }
1682024-11-21T23:10:00.971ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
1692024-11-21T23:10:00.992ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:325
1702024-11-21T23:10:01.005ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:407 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
1712024-11-21T23:10:01.005ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
1722024-11-21T23:10:01.005ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:419 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
1732024-11-21T23:10:01.005ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-11-21T23:10:01.007ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 127.547037ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f5879111-af20-4bcd-869c-07248f29867b", "content-length": "124", "date": "Thu, 21 Nov 2024 23:10:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f5879111-af20-4bcd-869c-07248f29867b" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-11-21T23:10:01.010ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:363
1762024-11-21T23:10:01.018ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1772024-11-21T23:10:01.041ZINFOSledAgent: successfully created encrypted zvol: name="oxi_67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1782024-11-21T23:10:01.041ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/swap" file = sled-agent/src/swap_device.rs:96
1792024-11-21T23:10:01.046ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:434 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
1802024-11-21T23:10:01.046ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1812024-11-21T23:10:01.046ZINFOSledAgent (BackingFs): Ensuring dataset oxi_67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/backing/fmd file = sled-agent/src/backing_fs.rs:138
1822024-11-21T23:10:01.135ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1832024-11-21T23:10:01.135ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-11-21T23:10:01.138ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 96.225846ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "28cc0327-dfbf-4b9b-aa7f-e7384680e538", "content-length": "124", "date": "Thu, 21 Nov 2024 23:10:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "28cc0327-dfbf-4b9b-aa7f-e7384680e538" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-11-21T23:10:01.235ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-11-21T23:10:01.236ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 93.41407ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c7e1b5e3-ed42-4c4d-974d-a3c716d63cfc", "content-length": "124", "date": "Thu, 21 Nov 2024 23:10:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c7e1b5e3-ed42-4c4d-974d-a3c716d63cfc" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-11-21T23:10:01.331ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-11-21T23:10:01.331ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 178.906671ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b5db006a-b1bc-4681-ac59-1d5580b90426", "content-length": "124", "date": "Thu, 21 Nov 2024 23:10:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b5db006a-b1bc-4681-ac59-1d5580b90426" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-11-21T23:10:01.354ZINFOSledAgent (BackingFs): Mounting oxi_67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1902024-11-21T23:10:01.367ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1912024-11-21T23:10:01.367ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1922024-11-21T23:10:01.367ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1932024-11-21T23:10:01.367ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1942024-11-21T23:10:01.511ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-11-21T23:10:01.512ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 287.644536ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d773474f-ba71-47cc-a72a-097885da1109", "content-length": "124", "date": "Thu, 21 Nov 2024 23:10:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d773474f-ba71-47cc-a72a-097885da1109" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-11-21T23:10:01.801ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1972024-11-21T23:10:01.802ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 260.001155ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0b57f4de-33de-490a-8d4a-9c2fedcdd1fd", "content-length": "124", "date": "Thu, 21 Nov 2024 23:10:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0b57f4de-33de-490a-8d4a-9c2fedcdd1fd" } file = clients/ddm-admin-client/src/lib.rs:177
1982024-11-21T23:10:02.063ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1992024-11-21T23:10:02.095ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2002024-11-21T23:10:02.101ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fd00:1122:3344:101::1]:64902 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2012024-11-21T23:10:02.101ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2022024-11-21T23:10:02.101ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:64902 file = oximeter/producer/src/lib.rs:279 interval = 30s producer_id = df02b17d-a4bd-4ac6-9447-5abe2628a534 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2032024-11-21T23:10:02.102ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 204.531873ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2042024-11-21T23:10:02.103ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2052024-11-21T23:10:02.103ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2062024-11-21T23:10:02.103ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:927 underlay_address = fd00:1122:3344:101::1
2072024-11-21T23:10:02.103ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
2082024-11-21T23:10:02.308ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 641.492307ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2092024-11-21T23:10:02.951ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 790.096596ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2102024-11-21T23:10:03.742ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.996907644s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2112024-11-21T23:10:04.717ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2676
2122024-11-21T23:10:04.717ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2921
2132024-11-21T23:10:04.717ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2546
2142024-11-21T23:10:04.717ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2593
2152024-11-21T23:10:04.717ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:3002
2162024-11-21T23:10:04.717ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3047
2172024-11-21T23:10:04.719ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2846
2182024-11-21T23:10:04.721ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2587
2192024-11-21T23:10:04.721ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='d7a26d40-543f-47f0-8255-7216f60ee965'/> <propval type="astring" name="rack_id" value='c1b8264e-5605-46dd-a3d8-c05674dfacdb'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:200::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe35:17d4'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:200::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="rack_id" value='c1b8264e-5605-46dd-a3d8-c05674dfacdb'/> <propval type="astring" name="sled_id" value='df02b17d-a4bd-4ac6-9447-5abe2628a534'/> <propval type="astring" name="sled_model" value='i86pc'/> <propval type="astring" name="sled_serial" value='sock'/> <propval type="astring" name="sled_revision" value='0'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='df02b17d-a4bd-4ac6-9447-5abe2628a534'/> <propval type="astring" name="rack_uuid" value='c1b8264e-5605-46dd-a3d8-c05674dfacdb'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='df02b17d-a4bd-4ac6-9447-5abe2628a534'/> <propval type="astring" name="rack_uuid" value='c1b8264e-5605-46dd-a3d8-c05674dfacdb'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2202024-11-21T23:10:04.721ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2212024-11-21T23:10:05.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 62.191971ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-11-21T23:10:05.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 75.213576ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-11-21T23:10:05.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 292.00796ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-11-21T23:10:05.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 501.172302ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-11-21T23:10:06.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 985.813726ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-11-21T23:10:06.742ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 3.667359106s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2272024-11-21T23:10:07.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.451088322s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2282024-11-21T23:10:09.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.20169408s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2292024-11-21T23:10:10.411ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.865636904s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2302024-11-21T23:10:10.972ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2312024-11-21T23:10:10.972ZINFOSledAgent (dropshot (Repo Depot)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fd00:1122:3344:101::1]:12348 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2322024-11-21T23:10:10.973ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:327 local_addr = [fd00:1122:3344:101::1]:12345
2332024-11-21T23:10:10.973ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2342024-11-21T23:10:10.973ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2352024-11-21T23:10:10.973ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:200
2362024-11-21T23:10:10.973ZINFOSledAgent (SledAgentMain): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/sled-agent-request.json"
2372024-11-21T23:10:10.973ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:200
2382024-11-21T23:10:10.974ZINFOSledAgent (SledAgentMain): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/sled-agent-request.json"
2392024-11-21T23:10:10.974ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2402024-11-21T23:10:10.974ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: Some(None), bgp_peers: [], autoneg: false, lldp: None, tx_eq: None }], bgp: [], bfd: [] })))
2412024-11-21T23:10:10.974ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:4005
2422024-11-21T23:10:10.974ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:200::1]:46004
2432024-11-21T23:10:11.161ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:200::1]:12346
2442024-11-21T23:10:11.161ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:200::1]:12346
2452024-11-21T23:10:11.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2462024-11-21T23:10:11.161ZINFOSledAgent (RSS): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/rss-service-plan-v5.json file = common/src/ledger.rs:189
2472024-11-21T23:10:11.161ZINFOSledAgent (RSS): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/rss-service-plan-v5.json file = common/src/ledger.rs:189
2482024-11-21T23:10:11.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2492024-11-21T23:10:11.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2502024-11-21T23:10:11.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2512024-11-21T23:10:11.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2522024-11-21T23:10:11.167ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4326 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2532024-11-21T23:10:11.167ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4342
2542024-11-21T23:10:11.167ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2552024-11-21T23:10:11.427ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57689
2562024-11-21T23:10:11.428ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2572024-11-21T23:10:11.428ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2582024-11-21T23:10:11.428ZINFOSledAgent (ServiceManager): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-zones.json file = common/src/ledger.rs:189
2592024-11-21T23:10:11.428ZINFOSledAgent (ServiceManager): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-zones.json file = common/src/ledger.rs:189
2602024-11-21T23:10:11.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 269 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57689 req_id = 56c8253c-d61c-4d49-aec2-453396384407 response_code = 200 uri = /inventory
2612024-11-21T23:10:11.460ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61984
2622024-11-21T23:10:11.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61984 req_id = 2320653c-6c3a-42d5-a55b-b9d469809d89 response_code = 200 uri = /sled-role
2632024-11-21T23:10:11.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
2642024-11-21T23:10:11.461ZINFOSledAgent (RSS): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/.rss-service-plan-v5.json.tmp file = common/src/ledger.rs:200
2652024-11-21T23:10:11.462ZINFOSledAgent (RSS): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/rss-service-plan-v5.json"
2662024-11-21T23:10:11.462ZINFOSledAgent (RSS): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/.rss-service-plan-v5.json.tmp file = common/src/ledger.rs:200
2672024-11-21T23:10:11.462ZINFOSledAgent (RSS): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/rss-service-plan-v5.json"
2682024-11-21T23:10:11.462ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:980
2692024-11-21T23:10:11.493ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: da5d01e2-22ea-4989-a7da-b44302e9ed18, pool_id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 60180f3b-296a-49fb-a3eb-db0529d85e18, pool_id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 28290b8e-4fb6-4153-b460-834e1a98721e, pool_id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 3288a0fb-dad8-4021-860d-9429ebb6197d, pool_id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: e857f345-58fb-438d-a07d-a43becfcfd7d, pool_id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 93d3ee9e-4d54-488f-a3e6-4699a4c8a5b7, pool_id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 21f7ea66-295c-4532-85e0-835391a4fd27, pool_id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: ba313001-226e-4c7c-964d-f5b4e3b4a86b, pool_id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 67b8c359-7627-4b69-aaf1-d558c2108883, pool_id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:382 sled_address = [fd00:1122:3344:101::1]:12345
2702024-11-21T23:10:11.493ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42314
2712024-11-21T23:10:11.494ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:884 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2722024-11-21T23:10:11.494ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: da5d01e2-22ea-4989-a7da-b44302e9ed18, pool_id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 60180f3b-296a-49fb-a3eb-db0529d85e18, pool_id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 28290b8e-4fb6-4153-b460-834e1a98721e, pool_id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 3288a0fb-dad8-4021-860d-9429ebb6197d, pool_id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: e857f345-58fb-438d-a07d-a43becfcfd7d, pool_id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 93d3ee9e-4d54-488f-a3e6-4699a4c8a5b7, pool_id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 21f7ea66-295c-4532-85e0-835391a4fd27, pool_id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: ba313001-226e-4c7c-964d-f5b4e3b4a86b, pool_id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 67b8c359-7627-4b69-aaf1-d558c2108883, pool_id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:603
2732024-11-21T23:10:11.494ZINFOSledAgent (StorageManager): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-physical-disks.json file = common/src/ledger.rs:189 request = omicron_physical_disks_ensure
2742024-11-21T23:10:11.494ZINFOSledAgent (StorageManager): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-physical-disks.json file = common/src/ledger.rs:189 request = omicron_physical_disks_ensure
2752024-11-21T23:10:11.494ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:1175 request = omicron_physical_disks_ensure
2762024-11-21T23:10:11.494ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2772024-11-21T23:10:11.494ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2782024-11-21T23:10:11.494ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2792024-11-21T23:10:11.494ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2802024-11-21T23:10:11.621ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2812024-11-21T23:10:11.621ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool)
2822024-11-21T23:10:12.074ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2832024-11-21T23:10:12.296ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2842024-11-21T23:10:12.297ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2852024-11-21T23:10:12.298ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
2862024-11-21T23:10:12.344ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }
2872024-11-21T23:10:12.352ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:240
2882024-11-21T23:10:12.352ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:242
2892024-11-21T23:10:12.352ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:246
2902024-11-21T23:10:12.352ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:248
2912024-11-21T23:10:12.352ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2922024-11-21T23:10:12.352ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2932024-11-21T23:10:12.352ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
2942024-11-21T23:10:12.388ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2952024-11-21T23:10:12.396ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone file = sled-storage/src/dataset.rs:303
2962024-11-21T23:10:12.574ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4 file = sled-storage/src/dataset.rs:389
2972024-11-21T23:10:12.582ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2982024-11-21T23:10:12.582ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2992024-11-21T23:10:12.583ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
3002024-11-21T23:10:12.583ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3012024-11-21T23:10:12.583ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
3022024-11-21T23:10:12.592ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
3032024-11-21T23:10:12.592ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool)
3042024-11-21T23:10:12.678ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }
3052024-11-21T23:10:12.686ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:240
3062024-11-21T23:10:12.686ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:242
3072024-11-21T23:10:12.686ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:246
3082024-11-21T23:10:12.686ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:248
3092024-11-21T23:10:12.686ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3102024-11-21T23:10:12.686ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3112024-11-21T23:10:12.686ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
3122024-11-21T23:10:12.723ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3132024-11-21T23:10:12.730ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone file = sled-storage/src/dataset.rs:303
3142024-11-21T23:10:12.899ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 file = sled-storage/src/dataset.rs:389
3152024-11-21T23:10:12.906ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3162024-11-21T23:10:12.906ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3172024-11-21T23:10:12.906ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3182024-11-21T23:10:12.906ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3192024-11-21T23:10:12.906ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3202024-11-21T23:10:12.916ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3212024-11-21T23:10:12.916ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool)
3222024-11-21T23:10:13.013ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }
3232024-11-21T23:10:13.021ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:240
3242024-11-21T23:10:13.021ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:242
3252024-11-21T23:10:13.021ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:246
3262024-11-21T23:10:13.021ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:248
3272024-11-21T23:10:13.022ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3282024-11-21T23:10:13.022ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3292024-11-21T23:10:13.022ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
3302024-11-21T23:10:13.057ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3312024-11-21T23:10:13.065ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone file = sled-storage/src/dataset.rs:303
3322024-11-21T23:10:13.077ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
3332024-11-21T23:10:13.240ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849 file = sled-storage/src/dataset.rs:389
3342024-11-21T23:10:13.248ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3352024-11-21T23:10:13.248ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3362024-11-21T23:10:13.248ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3372024-11-21T23:10:13.248ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3382024-11-21T23:10:13.248ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3392024-11-21T23:10:13.257ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3402024-11-21T23:10:13.257ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 60d88883-2ddc-48f7-92be-fb3130552398 (zpool)
3412024-11-21T23:10:13.348ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }
3422024-11-21T23:10:13.355ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:240
3432024-11-21T23:10:13.355ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:242
3442024-11-21T23:10:13.355ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:246
3452024-11-21T23:10:13.355ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:248
3462024-11-21T23:10:13.355ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3472024-11-21T23:10:13.355ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3482024-11-21T23:10:13.355ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
3492024-11-21T23:10:13.388ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3502024-11-21T23:10:13.395ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone file = sled-storage/src/dataset.rs:303
3512024-11-21T23:10:13.553ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_60d88883-2ddc-48f7-92be-fb3130552398 file = sled-storage/src/dataset.rs:389
3522024-11-21T23:10:13.559ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3532024-11-21T23:10:13.559ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3542024-11-21T23:10:13.559ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3552024-11-21T23:10:13.559ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3562024-11-21T23:10:13.559ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3572024-11-21T23:10:13.568ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3582024-11-21T23:10:13.568ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool)
3592024-11-21T23:10:13.655ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }
3602024-11-21T23:10:13.662ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:240
3612024-11-21T23:10:13.662ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:242
3622024-11-21T23:10:13.662ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:246
3632024-11-21T23:10:13.662ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:248
3642024-11-21T23:10:13.662ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3652024-11-21T23:10:13.662ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-11-21T23:10:13.662ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
3672024-11-21T23:10:13.697ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3682024-11-21T23:10:13.705ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone file = sled-storage/src/dataset.rs:303
3692024-11-21T23:10:13.863ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_56581dc2-c967-4930-aa2d-7f607fddd15e file = sled-storage/src/dataset.rs:389
3702024-11-21T23:10:13.869ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3712024-11-21T23:10:13.869ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3722024-11-21T23:10:13.869ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3732024-11-21T23:10:13.869ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3742024-11-21T23:10:13.869ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3752024-11-21T23:10:13.879ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3762024-11-21T23:10:13.879ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool)
3772024-11-21T23:10:13.968ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }
3782024-11-21T23:10:13.976ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:240
3792024-11-21T23:10:13.976ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:242
3802024-11-21T23:10:13.976ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:246
3812024-11-21T23:10:13.976ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:248
3822024-11-21T23:10:13.976ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3832024-11-21T23:10:13.976ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3842024-11-21T23:10:13.976ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
3852024-11-21T23:10:14.010ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3862024-11-21T23:10:14.018ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4a60e634-3d93-4efd-bc48-2e2658b8d062/crypt/zone file = sled-storage/src/dataset.rs:303
3872024-11-21T23:10:14.080ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
3882024-11-21T23:10:14.196ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4a60e634-3d93-4efd-bc48-2e2658b8d062 file = sled-storage/src/dataset.rs:389
3892024-11-21T23:10:14.204ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3902024-11-21T23:10:14.204ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3912024-11-21T23:10:14.204ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3922024-11-21T23:10:14.204ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3932024-11-21T23:10:14.204ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3942024-11-21T23:10:14.214ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3952024-11-21T23:10:14.214ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool)
3962024-11-21T23:10:14.308ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }
3972024-11-21T23:10:14.317ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:240
3982024-11-21T23:10:14.317ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:242
3992024-11-21T23:10:14.317ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:246
4002024-11-21T23:10:14.317ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:248
4012024-11-21T23:10:14.317ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
4022024-11-21T23:10:14.317ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
4032024-11-21T23:10:14.317ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
4042024-11-21T23:10:14.353ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4052024-11-21T23:10:14.361ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/zone file = sled-storage/src/dataset.rs:303
4062024-11-21T23:10:14.538ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2735b386-4b94-446b-b6e4-69280ab9e40a file = sled-storage/src/dataset.rs:389
4072024-11-21T23:10:14.546ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
4082024-11-21T23:10:14.546ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
4092024-11-21T23:10:14.546ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4102024-11-21T23:10:14.546ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4112024-11-21T23:10:14.546ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4122024-11-21T23:10:14.556ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4132024-11-21T23:10:14.556ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool)
4142024-11-21T23:10:14.651ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }
4152024-11-21T23:10:14.659ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:240
4162024-11-21T23:10:14.659ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:242
4172024-11-21T23:10:14.659ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:246
4182024-11-21T23:10:14.659ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:248
4192024-11-21T23:10:14.659ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4202024-11-21T23:10:14.659ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4212024-11-21T23:10:14.659ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
4222024-11-21T23:10:14.693ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4232024-11-21T23:10:14.702ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crypt/zone file = sled-storage/src/dataset.rs:303
4242024-11-21T23:10:14.891ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab file = sled-storage/src/dataset.rs:389
4252024-11-21T23:10:14.899ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4262024-11-21T23:10:14.899ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4272024-11-21T23:10:14.899ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4282024-11-21T23:10:14.899ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4292024-11-21T23:10:14.899ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4302024-11-21T23:10:14.910ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4312024-11-21T23:10:14.910ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool)
4322024-11-21T23:10:15.009ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:196 zpool = ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }
4332024-11-21T23:10:15.017ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:240
4342024-11-21T23:10:15.017ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:242
4352024-11-21T23:10:15.017ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:246
4362024-11-21T23:10:15.018ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:248
4372024-11-21T23:10:15.018ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4382024-11-21T23:10:15.018ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4392024-11-21T23:10:15.018ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:260
4402024-11-21T23:10:15.054ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4412024-11-21T23:10:15.063ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crypt/zone file = sled-storage/src/dataset.rs:303
4422024-11-21T23:10:15.081ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4432024-11-21T23:10:15.242ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 file = sled-storage/src/dataset.rs:389
4442024-11-21T23:10:15.249ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4452024-11-21T23:10:15.250ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4462024-11-21T23:10:15.250ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:200 request = omicron_physical_disks_ensure
4472024-11-21T23:10:15.250ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 67a39b8e-1dc3-40e2-8b0f-59c429bf4f91 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4c94aa86-6837-420b-a3c1-83899731e9b0 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4482024-11-21T23:10:15.250ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4492024-11-21T23:10:15.250ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4502024-11-21T23:10:15.250ZINFOSledAgent (StorageManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-physical-disks.json" request = omicron_physical_disks_ensure
4512024-11-21T23:10:15.250ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:200 request = omicron_physical_disks_ensure
4522024-11-21T23:10:15.250ZINFOSledAgent (StorageManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-physical-disks.json" request = omicron_physical_disks_ensure
4532024-11-21T23:10:15.250ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:888 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4542024-11-21T23:10:15.250ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
4552024-11-21T23:10:15.250ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:914 generation = Generation(1) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4562024-11-21T23:10:15.250ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:919 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4572024-11-21T23:10:15.250ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:924 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4582024-11-21T23:10:15.250ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:122 gen = Generation(1) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4592024-11-21T23:10:15.250ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:930 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4602024-11-21T23:10:15.250ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:799 gen = Generation(1)
4612024-11-21T23:10:15.250ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:935 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4622024-11-21T23:10:15.250ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 3757077 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42314 req_id = 52a5611b-b77f-403a-99e0-74f3d6bf2e76 response_code = 200 uri = /omicron-physical-disks
4632024-11-21T23:10:15.283ZINFOSledAgent (RSS): attempting to set up sled's datasets: DatasetsConfig { generation: Generation(1), datasets: {0646288e-6664-4bdf-a5b8-0cbecb247d11 (dataset): DatasetConfig { id: 0646288e-6664-4bdf-a5b8-0cbecb247d11 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_4fb64aa2-0fe1-47d4-b447-fc7e161481c6" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0686ebfa-21bb-4109-811d-839920f74823 (dataset): DatasetConfig { id: 0686ebfa-21bb-4109-811d-839920f74823 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1325c7a2-b88c-42d0-a166-3debdf30f7a2 (dataset): DatasetConfig { id: 1325c7a2-b88c-42d0-a166-3debdf30f7a2 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 17794c24-0720-4e81-80e3-60d82f305f0e (dataset): DatasetConfig { id: 17794c24-0720-4e81-80e3-60d82f305f0e (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_23f9a8fe-7b99-4a86-9e17-b09185836552" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1a17d0d5-7757-4b23-ab0d-80f680bf0068 (dataset): DatasetConfig { id: 1a17d0d5-7757-4b23-ab0d-80f680bf0068 (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 20783037-22f1-416c-8917-dac8b14521eb (dataset): DatasetConfig { id: 20783037-22f1-416c-8917-dac8b14521eb (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2811d626-f66e-4d13-bd5a-3a01ddfbfde4 (dataset): DatasetConfig { id: 2811d626-f66e-4d13-bd5a-3a01ddfbfde4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 28d69dfb-2f52-4937-b3c8-a6724fb9a8fd (dataset): DatasetConfig { id: 28d69dfb-2f52-4937-b3c8-a6724fb9a8fd (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2ab271d4-0db3-45b0-bae0-c99dd5d49fda (dataset): DatasetConfig { id: 2ab271d4-0db3-45b0-bae0-c99dd5d49fda (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2c6bb6c1-92bb-4dd8-9cd2-dc78d8b82cd8 (dataset): DatasetConfig { id: 2c6bb6c1-92bb-4dd8-9cd2-dc78d8b82cd8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_4a49f2d4-4a63-438e-9662-856d67052865" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 31d5e35b-4e3d-4dc9-a651-aa95fd6d7ae3 (dataset): DatasetConfig { id: 31d5e35b-4e3d-4dc9-a651-aa95fd6d7ae3 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 33570daf-d204-487c-a3dc-d340d67bba0b (dataset): DatasetConfig { id: 33570daf-d204-487c-a3dc-d340d67bba0b (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_4a05a39f-e9d2-455e-9cb5-f73902d6694b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 353f3312-9495-4467-81ee-23cd15a885a4 (dataset): DatasetConfig { id: 353f3312-9495-4467-81ee-23cd15a885a4 (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 359e87c3-8099-4007-9e6f-8d41294a1131 (dataset): DatasetConfig { id: 359e87c3-8099-4007-9e6f-8d41294a1131 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 36a1200b-6191-4278-a627-29c085136adc (dataset): DatasetConfig { id: 36a1200b-6191-4278-a627-29c085136adc (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3a07ae54-af6a-4508-a3c7-3803a37a1596 (dataset): DatasetConfig { id: 3a07ae54-af6a-4508-a3c7-3803a37a1596 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 41174fca-d60b-4637-bde3-6a806ae28192 (dataset): DatasetConfig { id: 41174fca-d60b-4637-bde3-6a806ae28192 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 43e11876-b574-4e96-97e4-46898d4fb2d6 (dataset): DatasetConfig { id: 43e11876-b574-4e96-97e4-46898d4fb2d6 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4477a59d-7801-420d-8c12-be7ef25d4730 (dataset): DatasetConfig { id: 4477a59d-7801-420d-8c12-be7ef25d4730 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_1a248290-007e-4b53-b1f1-6f7a9e3dbb71" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4844bd22-3782-406a-8d61-8603fefe283f (dataset): DatasetConfig { id: 4844bd22-3782-406a-8d61-8603fefe283f (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_9e92e318-bc88-437a-bf0e-cb6cc571e7b1" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 49482455-cdbe-4899-94bc-9d4244474e82 (dataset): DatasetConfig { id: 49482455-cdbe-4899-94bc-9d4244474e82 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_6ca40c88-6f5e-4727-8259-0993d419cff1" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4a53bade-703d-4a8e-b8ba-693884077454 (dataset): DatasetConfig { id: 4a53bade-703d-4a8e-b8ba-693884077454 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4ab918c0-5ef8-4404-9ede-17b689514f13 (dataset): DatasetConfig { id: 4ab918c0-5ef8-4404-9ede-17b689514f13 (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 4b04a7a7-55a3-457d-ae8b-4302aaa428ab (dataset): DatasetConfig { id: 4b04a7a7-55a3-457d-ae8b-4302aaa428ab (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_710d532b-120e-4fdc-9cae-a0252c955c30" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5cf204c8-c305-4d6f-89b2-9415ddc411e3 (dataset): DatasetConfig { id: 5cf204c8-c305-4d6f-89b2-9415ddc411e3 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_5267fc4d-d51f-4cd0-9820-0c103fa42eea" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6420ee84-a440-4ee2-9ee7-64cd807d04bd (dataset): DatasetConfig { id: 6420ee84-a440-4ee2-9ee7-64cd807d04bd (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_c5e898ed-4202-4777-9b7c-9d08a9bd3c3a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 64936e38-f87d-48f7-9fe9-a73f614a1c98 (dataset): DatasetConfig { id: 64936e38-f87d-48f7-9fe9-a73f614a1c98 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6b4cad43-4f31-4ee6-baf3-34133fb352f9 (dataset): DatasetConfig { id: 6b4cad43-4f31-4ee6-baf3-34133fb352f9 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_oximeter_13edfd02-a016-4abd-b9a3-b2af39dc85f9" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6e082723-189a-4ae8-ad1d-9c788e5ad902 (dataset): DatasetConfig { id: 6e082723-189a-4ae8-ad1d-9c788e5ad902 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 725d02e7-ef0b-45b6-9daf-400d782153b0 (dataset): DatasetConfig { id: 725d02e7-ef0b-45b6-9daf-400d782153b0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 7eac0f17-2a31-4c63-84f2-30ad51a8c1bc (dataset): DatasetConfig { id: 7eac0f17-2a31-4c63-84f2-30ad51a8c1bc (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7f63b141-11a6-4d53-8426-fe2d8c92530b (dataset): DatasetConfig { id: 7f63b141-11a6-4d53-8426-fe2d8c92530b (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 85304f96-cc43-4611-9315-f1ba7ead5e8b (dataset): DatasetConfig { id: 85304f96-cc43-4611-9315-f1ba7ead5e8b (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_8796224f-ce1e-4f10-8b1b-63b2661798e0" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 86128a35-b205-4776-b2f1-2fa5df01c4b9 (dataset): DatasetConfig { id: 86128a35-b205-4776-b2f1-2fa5df01c4b9 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_62b89797-eabf-4a54-ad84-6278e0d63bb2" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 87f7eea6-4e2b-4c9f-b8a7-1676f62d84ab (dataset): DatasetConfig { id: 87f7eea6-4e2b-4c9f-b8a7-1676f62d84ab (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8eb45c40-0d38-4946-828f-b00637a20c18 (dataset): DatasetConfig { id: 8eb45c40-0d38-4946-828f-b00637a20c18 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 96747428-6a4d-440e-90cd-3aa267ede24f (dataset): DatasetConfig { id: 96747428-6a4d-440e-90cd-3aa267ede24f (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 97fb4d68-ec04-4fff-9663-56b5a3dc9fae (dataset): DatasetConfig { id: 97fb4d68-ec04-4fff-9663-56b5a3dc9fae (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 983df6bd-7882-42d2-8191-f3afaae6e762 (dataset): DatasetConfig { id: 983df6bd-7882-42d2-8191-f3afaae6e762 (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_5d56a5dd-553d-43d1-ada2-7c96d8406842" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a1ff2dfd-9de8-46ee-b554-8b069dfb6df2 (dataset): DatasetConfig { id: a1ff2dfd-9de8-46ee-b554-8b069dfb6df2 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a4fd30f1-4698-41b3-b8fc-cbb83226684e (dataset): DatasetConfig { id: a4fd30f1-4698-41b3-b8fc-cbb83226684e (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_87ace5e0-9e49-47a5-9e60-8ef09cb36b5c" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a604712a-c938-4f89-b4b3-d4af9c23c9f8 (dataset): DatasetConfig { id: a604712a-c938-4f89-b4b3-d4af9c23c9f8 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_2a081fd5-0462-4328-9db6-071aae0d221b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b28a4670-0930-4cd4-b1e8-7bd1cc120b2b (dataset): DatasetConfig { id: b28a4670-0930-4cd4-b1e8-7bd1cc120b2b (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, b39c7a53-f320-4198-bafa-49bdea95c19a (dataset): DatasetConfig { id: b39c7a53-f320-4198-bafa-49bdea95c19a (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, b45db4e2-3947-420d-96cc-86dbebebf5e3 (dataset): DatasetConfig { id: b45db4e2-3947-420d-96cc-86dbebebf5e3 (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b858f569-1fe5-42ba-a647-e3f0a40a44e9 (dataset): DatasetConfig { id: b858f569-1fe5-42ba-a647-e3f0a40a44e9 (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, bc8bfb55-ea4b-4103-ab23-417c90b9a95b (dataset): DatasetConfig { id: bc8bfb55-ea4b-4103-ab23-417c90b9a95b (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, bda55e8e-d0c0-4309-bd71-c0a4c93a2ac9 (dataset): DatasetConfig { id: bda55e8e-d0c0-4309-bd71-c0a4c93a2ac9 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c01d9a75-c8ea-48b9-85a5-239afff33a22 (dataset): DatasetConfig { id: c01d9a75-c8ea-48b9-85a5-239afff33a22 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_e339542e-0ade-4538-9cba-05a7a02f2352" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c540951f-b03e-4936-a120-1d29ceab341d (dataset): DatasetConfig { id: c540951f-b03e-4936-a120-1d29ceab341d (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, c8467c06-cc64-41bb-a7dd-4bee12301547 (dataset): DatasetConfig { id: c8467c06-cc64-41bb-a7dd-4bee12301547 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_bac4f2f9-1ebe-4488-aa63-037e893be55f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, cd3d1292-cc02-4be2-86c8-bed0e140083e (dataset): DatasetConfig { id: cd3d1292-cc02-4be2-86c8-bed0e140083e (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_852e9877-8d05-4fca-8f0b-0d193b0ac883" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, cdd73160-8adf-4ae3-8284-d3a43d0bb817 (dataset): DatasetConfig { id: cdd73160-8adf-4ae3-8284-d3a43d0bb817 (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, cfda75af-330b-4e05-bb1c-ce25c890426b (dataset): DatasetConfig { id: cfda75af-330b-4e05-bb1c-ce25c890426b (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_c74c8efd-59c4-42a1-a06d-62bee6df73ab" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dafd3791-da6e-44fc-877f-acce507145d6 (dataset): DatasetConfig { id: dafd3791-da6e-44fc-877f-acce507145d6 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Clickhouse }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dc95eb60-7a86-4bec-8b0e-b0cc49c1c233 (dataset): DatasetConfig { id: dc95eb60-7a86-4bec-8b0e-b0cc49c1c233 (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dd1f66b6-28d5-4344-865a-a2d9989af950 (dataset): DatasetConfig { id: dd1f66b6-28d5-4344-865a-a2d9989af950 (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_6130c690-a32a-4ce5-a932-f89d5df30ab6" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e52fbe4a-10a9-4e7f-8673-73b6bf967338 (dataset): DatasetConfig { id: e52fbe4a-10a9-4e7f-8673-73b6bf967338 (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_54430eae-0e39-408d-a34d-a9a9a575ad61" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ee752ced-3eca-4e7c-943c-78b64aaa6135 (dataset): DatasetConfig { id: ee752ced-3eca-4e7c-943c-78b64aaa6135 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ef77125a-5142-4f6a-b05b-9e99a2ec57ca (dataset): DatasetConfig { id: ef77125a-5142-4f6a-b05b-9e99a2ec57ca (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fae30841-e06b-498a-af67-7a5ea9ae27b4 (dataset): DatasetConfig { id: fae30841-e06b-498a-af67-7a5ea9ae27b4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_clickhouse_ad64b46c-b548-42d7-a128-733f9dbdafda" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fb00fe6c-d12b-4efd-9abd-e4bd4caab4d2 (dataset): DatasetConfig { id: fb00fe6c-d12b-4efd-9abd-e4bd4caab4d2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_aa128c71-f260-4997-a503-cd00af8f00a9" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fcbf096a-f4db-4011-9059-2d8f549612a2 (dataset): DatasetConfig { id: fcbf096a-f4db-4011-9059-2d8f549612a2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fe74d525-d726-4e22-a010-cea47f9145d3 (dataset): DatasetConfig { id: fe74d525-d726-4e22-a010-cea47f9145d3 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ff36f9bd-2e05-47da-8076-7fec6f106100 (dataset): DatasetConfig { id: ff36f9bd-2e05-47da-8076-7fec6f106100 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, fff1e4cc-ae24-4f16-b0c7-a3b33393e85a (dataset): DatasetConfig { id: fff1e4cc-ae24-4f16-b0c7-a3b33393e85a (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }} } file = sled-agent/src/rack_setup/service.rs:457 sled_address = [fd00:1122:3344:101::1]:12345
4642024-11-21T23:10:15.283ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39695
4652024-11-21T23:10:15.284ZINFOSledAgent: datasets ensure file = sled-agent/src/sled_agent.rs:850 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4662024-11-21T23:10:15.284ZINFOSledAgent (StorageManager): Received DatasetsEnsure { config: DatasetsConfig { generation: Generation(1), datasets: {0646288e-6664-4bdf-a5b8-0cbecb247d11 (dataset): DatasetConfig { id: 0646288e-6664-4bdf-a5b8-0cbecb247d11 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_4fb64aa2-0fe1-47d4-b447-fc7e161481c6" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 0686ebfa-21bb-4109-811d-839920f74823 (dataset): DatasetConfig { id: 0686ebfa-21bb-4109-811d-839920f74823 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1325c7a2-b88c-42d0-a166-3debdf30f7a2 (dataset): DatasetConfig { id: 1325c7a2-b88c-42d0-a166-3debdf30f7a2 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 17794c24-0720-4e81-80e3-60d82f305f0e (dataset): DatasetConfig { id: 17794c24-0720-4e81-80e3-60d82f305f0e (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_23f9a8fe-7b99-4a86-9e17-b09185836552" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 1a17d0d5-7757-4b23-ab0d-80f680bf0068 (dataset): DatasetConfig { id: 1a17d0d5-7757-4b23-ab0d-80f680bf0068 (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 20783037-22f1-416c-8917-dac8b14521eb (dataset): DatasetConfig { id: 20783037-22f1-416c-8917-dac8b14521eb (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2811d626-f66e-4d13-bd5a-3a01ddfbfde4 (dataset): DatasetConfig { id: 2811d626-f66e-4d13-bd5a-3a01ddfbfde4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 28d69dfb-2f52-4937-b3c8-a6724fb9a8fd (dataset): DatasetConfig { id: 28d69dfb-2f52-4937-b3c8-a6724fb9a8fd (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2ab271d4-0db3-45b0-bae0-c99dd5d49fda (dataset): DatasetConfig { id: 2ab271d4-0db3-45b0-bae0-c99dd5d49fda (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 2c6bb6c1-92bb-4dd8-9cd2-dc78d8b82cd8 (dataset): DatasetConfig { id: 2c6bb6c1-92bb-4dd8-9cd2-dc78d8b82cd8 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_4a49f2d4-4a63-438e-9662-856d67052865" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 31d5e35b-4e3d-4dc9-a651-aa95fd6d7ae3 (dataset): DatasetConfig { id: 31d5e35b-4e3d-4dc9-a651-aa95fd6d7ae3 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 33570daf-d204-487c-a3dc-d340d67bba0b (dataset): DatasetConfig { id: 33570daf-d204-487c-a3dc-d340d67bba0b (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_4a05a39f-e9d2-455e-9cb5-f73902d6694b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 353f3312-9495-4467-81ee-23cd15a885a4 (dataset): DatasetConfig { id: 353f3312-9495-4467-81ee-23cd15a885a4 (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 359e87c3-8099-4007-9e6f-8d41294a1131 (dataset): DatasetConfig { id: 359e87c3-8099-4007-9e6f-8d41294a1131 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 36a1200b-6191-4278-a627-29c085136adc (dataset): DatasetConfig { id: 36a1200b-6191-4278-a627-29c085136adc (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 3a07ae54-af6a-4508-a3c7-3803a37a1596 (dataset): DatasetConfig { id: 3a07ae54-af6a-4508-a3c7-3803a37a1596 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 41174fca-d60b-4637-bde3-6a806ae28192 (dataset): DatasetConfig { id: 41174fca-d60b-4637-bde3-6a806ae28192 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 43e11876-b574-4e96-97e4-46898d4fb2d6 (dataset): DatasetConfig { id: 43e11876-b574-4e96-97e4-46898d4fb2d6 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4477a59d-7801-420d-8c12-be7ef25d4730 (dataset): DatasetConfig { id: 4477a59d-7801-420d-8c12-be7ef25d4730 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_1a248290-007e-4b53-b1f1-6f7a9e3dbb71" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4844bd22-3782-406a-8d61-8603fefe283f (dataset): DatasetConfig { id: 4844bd22-3782-406a-8d61-8603fefe283f (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_9e92e318-bc88-437a-bf0e-cb6cc571e7b1" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 49482455-cdbe-4899-94bc-9d4244474e82 (dataset): DatasetConfig { id: 49482455-cdbe-4899-94bc-9d4244474e82 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_6ca40c88-6f5e-4727-8259-0993d419cff1" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4a53bade-703d-4a8e-b8ba-693884077454 (dataset): DatasetConfig { id: 4a53bade-703d-4a8e-b8ba-693884077454 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 4ab918c0-5ef8-4404-9ede-17b689514f13 (dataset): DatasetConfig { id: 4ab918c0-5ef8-4404-9ede-17b689514f13 (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 4b04a7a7-55a3-457d-ae8b-4302aaa428ab (dataset): DatasetConfig { id: 4b04a7a7-55a3-457d-ae8b-4302aaa428ab (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_710d532b-120e-4fdc-9cae-a0252c955c30" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 5cf204c8-c305-4d6f-89b2-9415ddc411e3 (dataset): DatasetConfig { id: 5cf204c8-c305-4d6f-89b2-9415ddc411e3 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_5267fc4d-d51f-4cd0-9820-0c103fa42eea" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6420ee84-a440-4ee2-9ee7-64cd807d04bd (dataset): DatasetConfig { id: 6420ee84-a440-4ee2-9ee7-64cd807d04bd (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_c5e898ed-4202-4777-9b7c-9d08a9bd3c3a" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 64936e38-f87d-48f7-9fe9-a73f614a1c98 (dataset): DatasetConfig { id: 64936e38-f87d-48f7-9fe9-a73f614a1c98 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6b4cad43-4f31-4ee6-baf3-34133fb352f9 (dataset): DatasetConfig { id: 6b4cad43-4f31-4ee6-baf3-34133fb352f9 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_oximeter_13edfd02-a016-4abd-b9a3-b2af39dc85f9" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 6e082723-189a-4ae8-ad1d-9c788e5ad902 (dataset): DatasetConfig { id: 6e082723-189a-4ae8-ad1d-9c788e5ad902 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 725d02e7-ef0b-45b6-9daf-400d782153b0 (dataset): DatasetConfig { id: 725d02e7-ef0b-45b6-9daf-400d782153b0 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, 7eac0f17-2a31-4c63-84f2-30ad51a8c1bc (dataset): DatasetConfig { id: 7eac0f17-2a31-4c63-84f2-30ad51a8c1bc (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 7f63b141-11a6-4d53-8426-fe2d8c92530b (dataset): DatasetConfig { id: 7f63b141-11a6-4d53-8426-fe2d8c92530b (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 85304f96-cc43-4611-9315-f1ba7ead5e8b (dataset): DatasetConfig { id: 85304f96-cc43-4611-9315-f1ba7ead5e8b (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_8796224f-ce1e-4f10-8b1b-63b2661798e0" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 86128a35-b205-4776-b2f1-2fa5df01c4b9 (dataset): DatasetConfig { id: 86128a35-b205-4776-b2f1-2fa5df01c4b9 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_62b89797-eabf-4a54-ad84-6278e0d63bb2" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 87f7eea6-4e2b-4c9f-b8a7-1676f62d84ab (dataset): DatasetConfig { id: 87f7eea6-4e2b-4c9f-b8a7-1676f62d84ab (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 8eb45c40-0d38-4946-828f-b00637a20c18 (dataset): DatasetConfig { id: 8eb45c40-0d38-4946-828f-b00637a20c18 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 96747428-6a4d-440e-90cd-3aa267ede24f (dataset): DatasetConfig { id: 96747428-6a4d-440e-90cd-3aa267ede24f (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 97fb4d68-ec04-4fff-9663-56b5a3dc9fae (dataset): DatasetConfig { id: 97fb4d68-ec04-4fff-9663-56b5a3dc9fae (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, 983df6bd-7882-42d2-8191-f3afaae6e762 (dataset): DatasetConfig { id: 983df6bd-7882-42d2-8191-f3afaae6e762 (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_5d56a5dd-553d-43d1-ada2-7c96d8406842" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a1ff2dfd-9de8-46ee-b554-8b069dfb6df2 (dataset): DatasetConfig { id: a1ff2dfd-9de8-46ee-b554-8b069dfb6df2 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a4fd30f1-4698-41b3-b8fc-cbb83226684e (dataset): DatasetConfig { id: a4fd30f1-4698-41b3-b8fc-cbb83226684e (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_nexus_87ace5e0-9e49-47a5-9e60-8ef09cb36b5c" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, a604712a-c938-4f89-b4b3-d4af9c23c9f8 (dataset): DatasetConfig { id: a604712a-c938-4f89-b4b3-d4af9c23c9f8 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_2a081fd5-0462-4328-9db6-071aae0d221b" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b28a4670-0930-4cd4-b1e8-7bd1cc120b2b (dataset): DatasetConfig { id: b28a4670-0930-4cd4-b1e8-7bd1cc120b2b (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, b39c7a53-f320-4198-bafa-49bdea95c19a (dataset): DatasetConfig { id: b39c7a53-f320-4198-bafa-49bdea95c19a (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, b45db4e2-3947-420d-96cc-86dbebebf5e3 (dataset): DatasetConfig { id: b45db4e2-3947-420d-96cc-86dbebebf5e3 (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: Cockroach }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, b858f569-1fe5-42ba-a647-e3f0a40a44e9 (dataset): DatasetConfig { id: b858f569-1fe5-42ba-a647-e3f0a40a44e9 (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, bc8bfb55-ea4b-4103-ab23-417c90b9a95b (dataset): DatasetConfig { id: bc8bfb55-ea4b-4103-ab23-417c90b9a95b (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, bda55e8e-d0c0-4309-bd71-c0a4c93a2ac9 (dataset): DatasetConfig { id: bda55e8e-d0c0-4309-bd71-c0a4c93a2ac9 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: ExternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c01d9a75-c8ea-48b9-85a5-239afff33a22 (dataset): DatasetConfig { id: c01d9a75-c8ea-48b9-85a5-239afff33a22 (dataset), name: DatasetName { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }, kind: TransientZone { name: "oxz_cockroachdb_e339542e-0ade-4538-9cba-05a7a02f2352" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, c540951f-b03e-4936-a120-1d29ceab341d (dataset): DatasetConfig { id: c540951f-b03e-4936-a120-1d29ceab341d (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, c8467c06-cc64-41bb-a7dd-4bee12301547 (dataset): DatasetConfig { id: c8467c06-cc64-41bb-a7dd-4bee12301547 (dataset), name: DatasetName { pool_name: ZpoolName { id: 2735b386-4b94-446b-b6e4-69280ab9e40a (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_bac4f2f9-1ebe-4488-aa63-037e893be55f" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, cd3d1292-cc02-4be2-86c8-bed0e140083e (dataset): DatasetConfig { id: cd3d1292-cc02-4be2-86c8-bed0e140083e (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_852e9877-8d05-4fca-8f0b-0d193b0ac883" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, cdd73160-8adf-4ae3-8284-d3a43d0bb817 (dataset): DatasetConfig { id: cdd73160-8adf-4ae3-8284-d3a43d0bb817 (dataset), name: DatasetName { pool_name: ZpoolName { id: d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, cfda75af-330b-4e05-bb1c-ce25c890426b (dataset): DatasetConfig { id: cfda75af-330b-4e05-bb1c-ce25c890426b (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_external_dns_c74c8efd-59c4-42a1-a06d-62bee6df73ab" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dafd3791-da6e-44fc-877f-acce507145d6 (dataset): DatasetConfig { id: dafd3791-da6e-44fc-877f-acce507145d6 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Clickhouse }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dc95eb60-7a86-4bec-8b0e-b0cc49c1c233 (dataset): DatasetConfig { id: dc95eb60-7a86-4bec-8b0e-b0cc49c1c233 (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, dd1f66b6-28d5-4344-865a-a2d9989af950 (dataset): DatasetConfig { id: dd1f66b6-28d5-4344-865a-a2d9989af950 (dataset), name: DatasetName { pool_name: ZpoolName { id: e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_6130c690-a32a-4ce5-a932-f89d5df30ab6" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, e52fbe4a-10a9-4e7f-8673-73b6bf967338 (dataset): DatasetConfig { id: e52fbe4a-10a9-4e7f-8673-73b6bf967338 (dataset), name: DatasetName { pool_name: ZpoolName { id: 4a60e634-3d93-4efd-bc48-2e2658b8d062 (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_54430eae-0e39-408d-a34d-a9a9a575ad61" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ee752ced-3eca-4e7c-943c-78b64aaa6135 (dataset): DatasetConfig { id: ee752ced-3eca-4e7c-943c-78b64aaa6135 (dataset), name: DatasetName { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }, kind: TransientZone { name: "oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ef77125a-5142-4f6a-b05b-9e99a2ec57ca (dataset): DatasetConfig { id: ef77125a-5142-4f6a-b05b-9e99a2ec57ca (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZoneRoot }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fae30841-e06b-498a-af67-7a5ea9ae27b4 (dataset): DatasetConfig { id: fae30841-e06b-498a-af67-7a5ea9ae27b4 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: TransientZone { name: "oxz_clickhouse_ad64b46c-b548-42d7-a128-733f9dbdafda" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fb00fe6c-d12b-4efd-9abd-e4bd4caab4d2 (dataset): DatasetConfig { id: fb00fe6c-d12b-4efd-9abd-e4bd4caab4d2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: TransientZone { name: "oxz_crucible_pantry_aa128c71-f260-4997-a503-cd00af8f00a9" } }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fcbf096a-f4db-4011-9059-2d8f549612a2 (dataset): DatasetConfig { id: fcbf096a-f4db-4011-9059-2d8f549612a2 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, fe74d525-d726-4e22-a010-cea47f9145d3 (dataset): DatasetConfig { id: fe74d525-d726-4e22-a010-cea47f9145d3 (dataset), name: DatasetName { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }, kind: InternalDns }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }, ff36f9bd-2e05-47da-8076-7fec6f106100 (dataset): DatasetConfig { id: ff36f9bd-2e05-47da-8076-7fec6f106100 (dataset), name: DatasetName { pool_name: ZpoolName { id: 56581dc2-c967-4930-aa2d-7f607fddd15e (zpool), kind: External }, kind: Debug }, inner: SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } }, fff1e4cc-ae24-4f16-b0c7-a3b33393e85a (dataset): DatasetConfig { id: fff1e4cc-ae24-4f16-b0c7-a3b33393e85a (dataset), name: DatasetName { pool_name: ZpoolName { id: 60d88883-2ddc-48f7-92be-fb3130552398 (zpool), kind: External }, kind: Crucible }, inner: SharedDatasetConfig { compression: Off, quota: None, reservation: None } }} }, tx: ... } file = sled-storage/src/manager.rs:603
4672024-11-21T23:10:15.284ZINFOSledAgent (StorageManager): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-datasets.json file = common/src/ledger.rs:189 request = datasets_ensure
4682024-11-21T23:10:15.284ZINFOSledAgent (StorageManager): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-datasets.json file = common/src/ledger.rs:189 request = datasets_ensure
4692024-11-21T23:10:15.284ZINFOSledAgent (StorageManager): No previously-stored 'requested datasets', creating new ledger file = sled-storage/src/manager.rs:906 request = datasets_ensure
4702024-11-21T23:10:15.284ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4712024-11-21T23:10:15.284ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_crucible_4fb64aa2-0fe1-47d4-b447-fc7e161481c6"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_crucible_4fb64aa2-0fe1-47d4-b447-fc7e161481c6" } file = sled-storage/src/manager.rs:1338
4722024-11-21T23:10:15.387ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4732024-11-21T23:10:15.387ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crucible" } file = sled-storage/src/manager.rs:1338
4742024-11-21T23:10:15.477ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4752024-11-21T23:10:15.477ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1338
4762024-11-21T23:10:15.567ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4772024-11-21T23:10:15.567ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_cockroachdb_23f9a8fe-7b99-4a86-9e17-b09185836552"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_cockroachdb_23f9a8fe-7b99-4a86-9e17-b09185836552" } file = sled-storage/src/manager.rs:1338
4782024-11-21T23:10:15.667ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4792024-11-21T23:10:15.667ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/4a60e634-3d93-4efd-bc48-2e2658b8d062/crypt/zone"), full_name: "oxp_4a60e634-3d93-4efd-bc48-2e2658b8d062/crypt/zone" } file = sled-storage/src/manager.rs:1338
4802024-11-21T23:10:15.735ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4812024-11-21T23:10:15.736ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af" } file = sled-storage/src/manager.rs:1338
4822024-11-21T23:10:15.836ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4832024-11-21T23:10:15.836ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1338
4842024-11-21T23:10:15.926ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4852024-11-21T23:10:15.926ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7" } file = sled-storage/src/manager.rs:1338
4862024-11-21T23:10:16.028ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4872024-11-21T23:10:16.028ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crypt/zone"), full_name: "oxp_e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crypt/zone" } file = sled-storage/src/manager.rs:1338
4882024-11-21T23:10:16.086ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
4892024-11-21T23:10:16.097ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4902024-11-21T23:10:16.097ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_crucible_4a49f2d4-4a63-438e-9662-856d67052865"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_crucible_4a49f2d4-4a63-438e-9662-856d67052865" } file = sled-storage/src/manager.rs:1338
4912024-11-21T23:10:16.199ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4922024-11-21T23:10:16.199ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_2735b386-4b94-446b-b6e4-69280ab9e40a/crucible" } file = sled-storage/src/manager.rs:1338
4932024-11-21T23:10:16.288ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4942024-11-21T23:10:16.288ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/zone/oxz_nexus_4a05a39f-e9d2-455e-9cb5-f73902d6694b"), full_name: "oxp_2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/zone/oxz_nexus_4a05a39f-e9d2-455e-9cb5-f73902d6694b" } file = sled-storage/src/manager.rs:1338
4952024-11-21T23:10:16.389ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4962024-11-21T23:10:16.389ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crypt/debug"), full_name: "oxp_d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crypt/debug" } file = sled-storage/src/manager.rs:1338
4972024-11-21T23:10:16.458ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
4982024-11-21T23:10:16.458ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/internal_dns" } file = sled-storage/src/manager.rs:1338
4992024-11-21T23:10:16.552ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5002024-11-21T23:10:16.552ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone"), full_name: "oxp_60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone" } file = sled-storage/src/manager.rs:1338
5012024-11-21T23:10:16.620ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5022024-11-21T23:10:16.620ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone" } file = sled-storage/src/manager.rs:1338
5032024-11-21T23:10:16.690ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5042024-11-21T23:10:16.690ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7" } file = sled-storage/src/manager.rs:1338
5052024-11-21T23:10:16.787ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5062024-11-21T23:10:16.787ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crucible" } file = sled-storage/src/manager.rs:1338
5072024-11-21T23:10:16.875ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5082024-11-21T23:10:16.875ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_crucible_1a248290-007e-4b53-b1f1-6f7a9e3dbb71"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_crucible_1a248290-007e-4b53-b1f1-6f7a9e3dbb71" } file = sled-storage/src/manager.rs:1338
5092024-11-21T23:10:16.973ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5102024-11-21T23:10:16.973ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_cockroachdb_9e92e318-bc88-437a-bf0e-cb6cc571e7b1"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_cockroachdb_9e92e318-bc88-437a-bf0e-cb6cc571e7b1" } file = sled-storage/src/manager.rs:1338
5112024-11-21T23:10:17.072ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5122024-11-21T23:10:17.072ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_cockroachdb_6ca40c88-6f5e-4727-8259-0993d419cff1"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_cockroachdb_6ca40c88-6f5e-4727-8259-0993d419cff1" } file = sled-storage/src/manager.rs:1338
5132024-11-21T23:10:17.087ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
5142024-11-21T23:10:17.173ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5152024-11-21T23:10:17.173ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1338
5162024-11-21T23:10:17.265ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5172024-11-21T23:10:17.265ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/60d88883-2ddc-48f7-92be-fb3130552398/crypt/debug"), full_name: "oxp_60d88883-2ddc-48f7-92be-fb3130552398/crypt/debug" } file = sled-storage/src/manager.rs:1338
5182024-11-21T23:10:17.334ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5192024-11-21T23:10:17.334ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone/oxz_cockroachdb_710d532b-120e-4fdc-9cae-a0252c955c30"), full_name: "oxp_60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone/oxz_cockroachdb_710d532b-120e-4fdc-9cae-a0252c955c30" } file = sled-storage/src/manager.rs:1338
5202024-11-21T23:10:17.432ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5212024-11-21T23:10:17.432ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_external_dns_5267fc4d-d51f-4cd0-9820-0c103fa42eea"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_external_dns_5267fc4d-d51f-4cd0-9820-0c103fa42eea" } file = sled-storage/src/manager.rs:1338
5222024-11-21T23:10:17.532ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5232024-11-21T23:10:17.532ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone/oxz_crucible_c5e898ed-4202-4777-9b7c-9d08a9bd3c3a"), full_name: "oxp_60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone/oxz_crucible_c5e898ed-4202-4777-9b7c-9d08a9bd3c3a" } file = sled-storage/src/manager.rs:1338
5242024-11-21T23:10:17.631ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5252024-11-21T23:10:17.631ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1338
5262024-11-21T23:10:17.720ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5272024-11-21T23:10:17.720ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_oximeter_13edfd02-a016-4abd-b9a3-b2af39dc85f9"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_oximeter_13edfd02-a016-4abd-b9a3-b2af39dc85f9" } file = sled-storage/src/manager.rs:1338
5282024-11-21T23:10:17.819ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5292024-11-21T23:10:17.819ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/debug"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/debug" } file = sled-storage/src/manager.rs:1338
5302024-11-21T23:10:17.888ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5312024-11-21T23:10:17.888ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/debug"), full_name: "oxp_2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/debug" } file = sled-storage/src/manager.rs:1338
5322024-11-21T23:10:17.955ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5332024-11-21T23:10:17.956ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crucible" } file = sled-storage/src/manager.rs:1338
5342024-11-21T23:10:18.044ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5352024-11-21T23:10:18.044ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/external_dns" } file = sled-storage/src/manager.rs:1338
5362024-11-21T23:10:18.090ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
5372024-11-21T23:10:18.135ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5382024-11-21T23:10:18.135ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crypt/zone/oxz_crucible_8796224f-ce1e-4f10-8b1b-63b2661798e0"), full_name: "oxp_d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crypt/zone/oxz_crucible_8796224f-ce1e-4f10-8b1b-63b2661798e0" } file = sled-storage/src/manager.rs:1338
5392024-11-21T23:10:18.234ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5402024-11-21T23:10:18.234ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_crucible_pantry_62b89797-eabf-4a54-ad84-6278e0d63bb2"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_crucible_pantry_62b89797-eabf-4a54-ad84-6278e0d63bb2" } file = sled-storage/src/manager.rs:1338
5412024-11-21T23:10:18.334ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5422024-11-21T23:10:18.334ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone" } file = sled-storage/src/manager.rs:1338
5432024-11-21T23:10:18.401ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5442024-11-21T23:10:18.401ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/zone"), full_name: "oxp_2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/zone" } file = sled-storage/src/manager.rs:1338
5452024-11-21T23:10:18.470ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5462024-11-21T23:10:18.470ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/internal_dns" } file = sled-storage/src/manager.rs:1338
5472024-11-21T23:10:18.563ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5482024-11-21T23:10:18.563ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone" } file = sled-storage/src/manager.rs:1338
5492024-11-21T23:10:18.630ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5502024-11-21T23:10:18.630ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone/oxz_nexus_5d56a5dd-553d-43d1-ada2-7c96d8406842"), full_name: "oxp_60d88883-2ddc-48f7-92be-fb3130552398/crypt/zone/oxz_nexus_5d56a5dd-553d-43d1-ada2-7c96d8406842" } file = sled-storage/src/manager.rs:1338
5512024-11-21T23:10:18.729ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5522024-11-21T23:10:18.730ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crucible" } file = sled-storage/src/manager.rs:1338
5532024-11-21T23:10:18.816ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5542024-11-21T23:10:18.816ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_nexus_87ace5e0-9e49-47a5-9e60-8ef09cb36b5c"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_nexus_87ace5e0-9e49-47a5-9e60-8ef09cb36b5c" } file = sled-storage/src/manager.rs:1338
5552024-11-21T23:10:18.913ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5562024-11-21T23:10:18.913ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_crucible_2a081fd5-0462-4328-9db6-071aae0d221b"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_crucible_2a081fd5-0462-4328-9db6-071aae0d221b" } file = sled-storage/src/manager.rs:1338
5572024-11-21T23:10:19.012ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5582024-11-21T23:10:19.012ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/debug"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/debug" } file = sled-storage/src/manager.rs:1338
5592024-11-21T23:10:19.080ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5602024-11-21T23:10:19.080ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/debug"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/debug" } file = sled-storage/src/manager.rs:1338
5612024-11-21T23:10:19.091ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
5622024-11-21T23:10:19.149ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5632024-11-21T23:10:19.149ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_60d88883-2ddc-48f7-92be-fb3130552398/crypt/cockroachdb" } file = sled-storage/src/manager.rs:1338
5642024-11-21T23:10:19.240ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5652024-11-21T23:10:19.240ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crypt/debug"), full_name: "oxp_e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crypt/debug" } file = sled-storage/src/manager.rs:1338
5662024-11-21T23:10:19.307ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5672024-11-21T23:10:19.307ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_4a60e634-3d93-4efd-bc48-2e2658b8d062/crucible" } file = sled-storage/src/manager.rs:1338
5682024-11-21T23:10:19.394ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5692024-11-21T23:10:19.394ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/external_dns" } file = sled-storage/src/manager.rs:1338
5702024-11-21T23:10:19.483ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5712024-11-21T23:10:19.483ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_cockroachdb_e339542e-0ade-4538-9cba-05a7a02f2352"), full_name: "oxp_bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4/crypt/zone/oxz_cockroachdb_e339542e-0ade-4538-9cba-05a7a02f2352" } file = sled-storage/src/manager.rs:1338
5722024-11-21T23:10:19.581ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5732024-11-21T23:10:19.581ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/4a60e634-3d93-4efd-bc48-2e2658b8d062/crypt/debug"), full_name: "oxp_4a60e634-3d93-4efd-bc48-2e2658b8d062/crypt/debug" } file = sled-storage/src/manager.rs:1338
5742024-11-21T23:10:19.649ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5752024-11-21T23:10:19.649ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/zone/oxz_crucible_bac4f2f9-1ebe-4488-aa63-037e893be55f"), full_name: "oxp_2735b386-4b94-446b-b6e4-69280ab9e40a/crypt/zone/oxz_crucible_bac4f2f9-1ebe-4488-aa63-037e893be55f" } file = sled-storage/src/manager.rs:1338
5762024-11-21T23:10:19.748ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5772024-11-21T23:10:19.748ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_crucible_pantry_852e9877-8d05-4fca-8f0b-0d193b0ac883"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_crucible_pantry_852e9877-8d05-4fca-8f0b-0d193b0ac883" } file = sled-storage/src/manager.rs:1338
5782024-11-21T23:10:19.947ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5792024-11-21T23:10:19.947ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crypt/zone"), full_name: "oxp_d7f8995a-74b6-4a9d-b142-4b7ddbdf63e1/crypt/zone" } file = sled-storage/src/manager.rs:1338
5802024-11-21T23:10:20.044ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5812024-11-21T23:10:20.044ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_external_dns_c74c8efd-59c4-42a1-a06d-62bee6df73ab"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_external_dns_c74c8efd-59c4-42a1-a06d-62bee6df73ab" } file = sled-storage/src/manager.rs:1338
5822024-11-21T23:10:20.093ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
5832024-11-21T23:10:20.198ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5842024-11-21T23:10:20.198ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/clickhouse" } file = sled-storage/src/manager.rs:1338
5852024-11-21T23:10:20.332ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5862024-11-21T23:10:20.332ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crucible" } file = sled-storage/src/manager.rs:1338
5872024-11-21T23:10:20.471ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5882024-11-21T23:10:20.471ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crypt/zone/oxz_crucible_6130c690-a32a-4ce5-a932-f89d5df30ab6"), full_name: "oxp_e5bb58f6-a9bf-4899-acd4-a32e9b6d18ab/crypt/zone/oxz_crucible_6130c690-a32a-4ce5-a932-f89d5df30ab6" } file = sled-storage/src/manager.rs:1338
5892024-11-21T23:10:20.609ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5902024-11-21T23:10:20.609ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/4a60e634-3d93-4efd-bc48-2e2658b8d062/crypt/zone/oxz_crucible_54430eae-0e39-408d-a34d-a9a9a575ad61"), full_name: "oxp_4a60e634-3d93-4efd-bc48-2e2658b8d062/crypt/zone/oxz_crucible_54430eae-0e39-408d-a34d-a9a9a575ad61" } file = sled-storage/src/manager.rs:1338
5912024-11-21T23:10:20.740ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5922024-11-21T23:10:20.740ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e"), full_name: "oxp_e858b5e9-aa1b-415e-9d0d-1b75991b7849/crypt/zone/oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e" } file = sled-storage/src/manager.rs:1338
5932024-11-21T23:10:20.874ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5942024-11-21T23:10:20.875ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone" } file = sled-storage/src/manager.rs:1338
5952024-11-21T23:10:20.944ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2919013623316299948 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
5962024-11-21T23:10:20.945ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5126378412384038234 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
5972024-11-21T23:10:20.964ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
5982024-11-21T23:10:20.964ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_clickhouse_ad64b46c-b548-42d7-a128-733f9dbdafda"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/zone/oxz_clickhouse_ad64b46c-b548-42d7-a128-733f9dbdafda" } file = sled-storage/src/manager.rs:1338
5992024-11-21T23:10:21.043ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4350
6002024-11-21T23:10:21.044ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4360 gateway = fd00:1122:3344:101::1
6012024-11-21T23:10:21.050ZINFOSledAgent (ServiceManager): Default route already exists file = sled-agent/src/services.rs:4374 gateway = fd00:1122:3344:101::1
6022024-11-21T23:10:21.050ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4439
6032024-11-21T23:10:21.098ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6042024-11-21T23:10:21.116ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
6052024-11-21T23:10:21.116ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_crucible_pantry_aa128c71-f260-4997-a503-cd00af8f00a9"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/zone/oxz_crucible_pantry_aa128c71-f260-4997-a503-cd00af8f00a9" } file = sled-storage/src/manager.rs:1338
6062024-11-21T23:10:21.196ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4505
6072024-11-21T23:10:21.196ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4532
6082024-11-21T23:10:21.238ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4545
6092024-11-21T23:10:21.238ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4391
6102024-11-21T23:10:21.263ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
6112024-11-21T23:10:21.263ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crucible" } file = sled-storage/src/manager.rs:1338
6122024-11-21T23:10:21.278ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 12.897827825s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:387 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6132024-11-21T23:10:21.293ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4433
6142024-11-21T23:10:21.293ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4512 rack_subnet = fd00:1122:3344::
6152024-11-21T23:10:21.314ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4523
6162024-11-21T23:10:21.314ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4565
6172024-11-21T23:10:21.385ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4597
6182024-11-21T23:10:21.385ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4603
6192024-11-21T23:10:21.400ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
6202024-11-21T23:10:21.400ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_7114ae1a-4fea-4be4-be38-9a0f250ad6f4/crypt/internal_dns" } file = sled-storage/src/manager.rs:1338
6212024-11-21T23:10:21.476ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4644
6222024-11-21T23:10:21.476ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
6232024-11-21T23:10:21.514ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/nexus.rs:530 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6242024-11-21T23:10:21.517ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:418
6252024-11-21T23:10:21.518ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/nexus.rs:530 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6262024-11-21T23:10:21.536ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
6272024-11-21T23:10:21.536ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: GzipN { level: GzipLevel(9) }, quota: Some(ByteCount(107374182400)), reservation: None } details = DatasetCreationDetails { zoned: false, mountpoint: Path("/pool/ext/56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/debug"), full_name: "oxp_56581dc2-c967-4930-aa2d-7f607fddd15e/crypt/debug" } file = sled-storage/src/manager.rs:1338
6282024-11-21T23:10:21.556ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:686
6292024-11-21T23:10:21.556ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:731
6302024-11-21T23:10:21.557ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:734 uptime = 1
6312024-11-21T23:10:21.557ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: Some(\n None,\n ),\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n tx_eq: None,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:440
6322024-11-21T23:10:21.606ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4177
6332024-11-21T23:10:21.629ZINFOSledAgent (StorageManager): Ensuring dataset file = sled-storage/src/manager.rs:952 request = datasets_ensure
6342024-11-21T23:10:21.629ZINFOSledAgent (StorageManager): ensure_dataset config = SharedDatasetConfig { compression: Off, quota: None, reservation: None } details = DatasetCreationDetails { zoned: true, mountpoint: Path("/data"), full_name: "oxp_60d88883-2ddc-48f7-92be-fb3130552398/crucible" } file = sled-storage/src/manager.rs:1338
6352024-11-21T23:10:21.757ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/.omicron-datasets.json.tmp file = common/src/ledger.rs:200 request = datasets_ensure
6362024-11-21T23:10:21.757ZINFOSledAgent (StorageManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-datasets.json" request = datasets_ensure
6372024-11-21T23:10:21.757ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/.omicron-datasets.json.tmp file = common/src/ledger.rs:200 request = datasets_ensure
6382024-11-21T23:10:21.757ZINFOSledAgent (StorageManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-datasets.json" request = datasets_ensure
6392024-11-21T23:10:21.758ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:603
6402024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
6412024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
6422024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
6432024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
6442024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
6452024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
6462024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
6472024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
6482024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
6492024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
6502024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
6512024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
6522024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
6532024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
6542024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
6552024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
6562024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
6572024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
6582024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
6592024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
6602024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
6612024-11-21T23:10:21.758ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
6622024-11-21T23:10:21.758ZINFOSledAgent: datasets ensure: Updated storage file = sled-agent/src/sled_agent.rs:852 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6632024-11-21T23:10:21.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 6474306 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:39695 req_id = 7c38eae3-289d-4638-9a74-75e955b590b7 response_code = 200 uri = /datasets
6642024-11-21T23:10:21.791ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 820116a3-d9c1-4adb-873f-47498c0c38af (service), filesystem_pool: Some(ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 34c52a54-f90d-4998-a35f-4abe932744d7 (service), filesystem_pool: Some(ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 32d08044-1c99-436c-a7a8-db8e905f59e7 (service), filesystem_pool: Some(ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:538 sled_address = [fd00:1122:3344:101::1]:12345
6652024-11-21T23:10:21.791ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55279
6662024-11-21T23:10:21.791ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
6672024-11-21T23:10:21.791ZINFOSledAgent (ServiceManager): No ledger in /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-zones.json file = common/src/ledger.rs:189
6682024-11-21T23:10:21.791ZINFOSledAgent (ServiceManager): No ledger in /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-zones.json file = common/src/ledger.rs:189
6692024-11-21T23:10:21.791ZINFOSledAgent (StorageManager): Received DatasetsConfigList { tx: ... } file = sled-storage/src/manager.rs:603
6702024-11-21T23:10:21.791ZINFOSledAgent (StorageManager): Reading ledger from /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-datasets.json file = common/src/ledger.rs:178 request = datasets_config_list
6712024-11-21T23:10:21.791ZINFOSledAgent (StorageManager): Reading ledger from /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-datasets.json file = common/src/ledger.rs:178 request = datasets_config_list
6722024-11-21T23:10:21.791ZINFOSledAgent (StorageManager): Found ledger on internal storage file = sled-storage/src/manager.rs:992 request = datasets_config_list
6732024-11-21T23:10:21.792ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
6742024-11-21T23:10:21.836ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
6752024-11-21T23:10:21.836ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
6762024-11-21T23:10:21.836ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
6772024-11-21T23:10:21.886ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7 file = illumos-utils/src/zone.rs:325
6782024-11-21T23:10:21.890ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af file = illumos-utils/src/zone.rs:325
6792024-11-21T23:10:21.893ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7 file = illumos-utils/src/zone.rs:325
6802024-11-21T23:10:21.905ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7 file = illumos-utils/src/zone.rs:363
6812024-11-21T23:10:21.910ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af file = illumos-utils/src/zone.rs:363
6822024-11-21T23:10:21.914ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7 file = illumos-utils/src/zone.rs:363
6832024-11-21T23:10:22.100ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/df02b17d-a4bd-4ac6-9447-5abe2628a534) file = sled-agent/src/probe_manager.rs:231 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6842024-11-21T23:10:22.300ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6852024-11-21T23:10:22.300ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
6862024-11-21T23:10:23.517ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6872024-11-21T23:10:23.517ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
6882024-11-21T23:10:23.518ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
6892024-11-21T23:10:24.004ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6902024-11-21T23:10:24.482ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6912024-11-21T23:10:24.482ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
6922024-11-21T23:10:24.482ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
6932024-11-21T23:10:24.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 39.021272ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
6942024-11-21T23:10:24.491ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
6952024-11-21T23:10:24.495ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
6962024-11-21T23:10:24.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 124.21638ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
6972024-11-21T23:10:24.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 26.1756ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
6982024-11-21T23:10:24.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 43.887595ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
6992024-11-21T23:10:24.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 136.789524ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7002024-11-21T23:10:24.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 158.551508ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7012024-11-21T23:10:24.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 61.838308ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7022024-11-21T23:10:24.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 250.12968ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7032024-11-21T23:10:25.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 444.451039ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7042024-11-21T23:10:25.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 208.317613ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7052024-11-21T23:10:25.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 211.827618ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7062024-11-21T23:10:25.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 516.479456ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7072024-11-21T23:10:25.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 746.487162ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7082024-11-21T23:10:25.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 1.07586158s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7092024-11-21T23:10:25.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 1.161320863s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7102024-11-21T23:10:26.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 1.488956176s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7112024-11-21T23:10:26.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 594.004998ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7122024-11-21T23:10:26.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 570.453246ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7132024-11-21T23:10:27.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 1.298285377s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7142024-11-21T23:10:27.541ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 1.0586591s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7152024-11-21T23:10:27.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 879.697538ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7162024-11-21T23:10:28.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 782.526774ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7172024-11-21T23:10:28.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 1.049641327s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7182024-11-21T23:10:28.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 1.098118686s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7192024-11-21T23:10:29.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7") failed: Property not found. retry in 1.365018532s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7202024-11-21T23:10:29.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7") failed: Property not found. retry in 1.15728784s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7212024-11-21T23:10:29.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af") failed: Property not found. retry in 1.068020006s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7222024-11-21T23:10:30.814ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3318 zone = oxz_internal_dns_34c52a54-f90d-4998-a35f-4abe932744d7
7232024-11-21T23:10:30.814ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3318 zone = oxz_internal_dns_820116a3-d9c1-4adb-873f-47498c0c38af
7242024-11-21T23:10:30.814ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3318 zone = oxz_internal_dns_32d08044-1c99-436c-a7a8-db8e905f59e7
7252024-11-21T23:10:30.814ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/.omicron-zones.json.tmp file = common/src/ledger.rs:200
7262024-11-21T23:10:30.814ZINFOSledAgent (ServiceManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-zones.json"
7272024-11-21T23:10:30.814ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/.omicron-zones.json.tmp file = common/src/ledger.rs:200
7282024-11-21T23:10:30.815ZINFOSledAgent (ServiceManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-zones.json"
7292024-11-21T23:10:30.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9023899 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55279 req_id = 34944634-971c-45ff-aa7a-af758e8f4d09 response_code = 204 uri = /omicron-zones
7302024-11-21T23:10:30.818ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:655
7312024-11-21T23:10:30.946ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2919013623316299948 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7322024-11-21T23:10:30.946ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5126378412384038234 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7332024-11-21T23:10:32.021ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:655
7342024-11-21T23:10:32.096ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:655
7352024-11-21T23:10:32.169ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:691
7362024-11-21T23:10:32.169ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
7372024-11-21T23:10:32.200ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
7382024-11-21T23:10:32.201ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
7392024-11-21T23:10:32.201ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
7402024-11-21T23:10:32.231ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 820116a3-d9c1-4adb-873f-47498c0c38af (service), filesystem_pool: Some(ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7114ae1a-4fea-4be4-be38-9a0f250ad6f4 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 34c52a54-f90d-4998-a35f-4abe932744d7 (service), filesystem_pool: Some(ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bc8b77fc-ded9-497e-9dd4-28fb6a7b1fb4 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 32d08044-1c99-436c-a7a8-db8e905f59e7 (service), filesystem_pool: Some(ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 71e73077-bc8e-4f9f-8689-878cda8ba43e (service), filesystem_pool: Some(ZpoolName { id: e858b5e9-aa1b-415e-9d0d-1b75991b7849 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b11bb979-bfa7-42d1-abb4-110ef60451a7, kind: Service { id: 71e73077-bc8e-4f9f-8689-878cda8ba43e }, name: Name("ntp-71e73077-bc8e-4f9f-8689-878cda8ba43e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 44])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:538 sled_address = [fd00:1122:3344:101::1]:12345
7412024-11-21T23:10:32.231ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59783
7422024-11-21T23:10:32.231ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
7432024-11-21T23:10:32.231ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-zones.json file = common/src/ledger.rs:178
7442024-11-21T23:10:32.231ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-zones.json file = common/src/ledger.rs:178
7452024-11-21T23:10:32.231ZINFOSledAgent (StorageManager): Received DatasetsConfigList { tx: ... } file = sled-storage/src/manager.rs:603
7462024-11-21T23:10:32.231ZINFOSledAgent (StorageManager): Reading ledger from /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-datasets.json file = common/src/ledger.rs:178 request = datasets_config_list
7472024-11-21T23:10:32.231ZINFOSledAgent (StorageManager): Reading ledger from /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-datasets.json file = common/src/ledger.rs:178 request = datasets_config_list
7482024-11-21T23:10:32.231ZINFOSledAgent (StorageManager): Found ledger on internal storage file = sled-storage/src/manager.rs:992 request = datasets_config_list
7492024-11-21T23:10:32.231ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
7502024-11-21T23:10:32.245ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
7512024-11-21T23:10:32.279ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
7522024-11-21T23:10:32.279ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
7532024-11-21T23:10:32.279ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
7542024-11-21T23:10:32.302ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7552024-11-21T23:10:32.303ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7562024-11-21T23:10:32.309ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 147, 44]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
7572024-11-21T23:10:32.309ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1271 zone_type = boundary_ntp
7582024-11-21T23:10:32.309ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
7592024-11-21T23:10:32.309ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
7602024-11-21T23:10:32.310ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
7612024-11-21T23:10:32.310ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:603
7622024-11-21T23:10:32.334ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e file = illumos-utils/src/zone.rs:325
7632024-11-21T23:10:32.352ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e file = illumos-utils/src/zone.rs:363
7642024-11-21T23:10:32.977ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7652024-11-21T23:10:32.977ZINFOSledAgent (ServiceManager): Booting oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7662024-11-21T23:10:33.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 35.165544ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7672024-11-21T23:10:33.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 61.505553ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7682024-11-21T23:10:33.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 278.196134ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7692024-11-21T23:10:33.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 241.548094ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7702024-11-21T23:10:33.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 794.809744ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7712024-11-21T23:10:34.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 961.803412ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7722024-11-21T23:10:35.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 556.42732ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7732024-11-21T23:10:36.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 882.640032ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7742024-11-21T23:10:37.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 870.561485ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7752024-11-21T23:10:37.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 564.733808ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7762024-11-21T23:10:38.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 610.306145ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7772024-11-21T23:10:39.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e") failed: Property not found. retry in 614.551044ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7782024-11-21T23:10:39.708ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3318 zone = oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e
7792024-11-21T23:10:39.708ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/.omicron-zones.json.tmp file = common/src/ledger.rs:200
7802024-11-21T23:10:39.708ZINFOSledAgent (ServiceManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/67a39b8e-1dc3-40e2-8b0f-59c429bf4f91/config/omicron-zones.json"
7812024-11-21T23:10:39.708ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/.omicron-zones.json.tmp file = common/src/ledger.rs:200
7822024-11-21T23:10:39.709ZINFOSledAgent (ServiceManager): Successfully wrote to ledger file = common/src/ledger.rs:133 path = "/pool/int/4c94aa86-6837-420b-a3c1-83899731e9b0/config/omicron-zones.json"
7832024-11-21T23:10:39.709ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7477873 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59783 req_id = 36b6c56d-9a7c-4991-a431-18e9e4b36947 response_code = 204 uri = /omicron-zones
7842024-11-21T23:10:39.711ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:732
7852024-11-21T23:10:39.742ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
7862024-11-21T23:10:39.742ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62845
7872024-11-21T23:10:39.751ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3958
7882024-11-21T23:10:39.751ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 8983 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62845 req_id = f868388e-d7c6-4bd9-8156-a9165bd61975 response_code = 500 uri = /timesync
7892024-11-21T23:10:39.754ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
7902024-11-21T23:10:39.916ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
7912024-11-21T23:10:39.916ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37041
7922024-11-21T23:10:40.647ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5398698369582487023 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7932024-11-21T23:10:40.812ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15858676633825570490 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7942024-11-21T23:10:40.816ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2459447259745793341 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7952024-11-21T23:10:40.947ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2919013623316299948 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7962024-11-21T23:10:40.947ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5126378412384038234 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7972024-11-21T23:10:42.305ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7982024-11-21T23:10:42.305ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
7992024-11-21T23:10:43.925ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3958
8002024-11-21T23:10:43.925ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 4009104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37041 req_id = 5a632b47-80d6-438f-8258-b70940845120 response_code = 500 uri = /timesync
8012024-11-21T23:10:43.928ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8022024-11-21T23:10:44.048ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8032024-11-21T23:10:44.048ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54879
8042024-11-21T23:10:44.057ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3958
8052024-11-21T23:10:44.057ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 8766 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54879 req_id = 46559ed5-a03f-47fd-b980-556fefa183d7 response_code = 500 uri = /timesync
8062024-11-21T23:10:44.061ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8072024-11-21T23:10:44.187ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8082024-11-21T23:10:44.187ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52517
8092024-11-21T23:10:48.195ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3958
8102024-11-21T23:10:48.196ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 4008474 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52517 req_id = 6db0970d-cef1-427a-bca3-e2f5112804b8 response_code = 500 uri = /timesync
8112024-11-21T23:10:48.199ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8122024-11-21T23:10:48.321ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8132024-11-21T23:10:48.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53316
8142024-11-21T23:10:48.331ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9090 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53316 req_id = 4a5ad9b6-b8c7-40d3-b4a8-9919f741b0e0 response_code = 200 uri = /timesync
8152024-11-21T23:10:48.334ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8162024-11-21T23:10:48.334ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8172024-11-21T23:10:48.495ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8182024-11-21T23:10:48.495ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57812
8192024-11-21T23:10:48.504ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8351 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57812 req_id = 9fcc6454-8954-47f2-97ab-8fb65d03b6b1 response_code = 200 uri = /timesync
8202024-11-21T23:10:48.507ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8212024-11-21T23:10:48.507ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8222024-11-21T23:10:48.827ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8232024-11-21T23:10:48.828ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35679
8242024-11-21T23:10:48.836ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7992 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35679 req_id = 4374df11-8ce4-4b69-8607-fe67ba47682b response_code = 200 uri = /timesync
8252024-11-21T23:10:48.838ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8262024-11-21T23:10:48.838ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8272024-11-21T23:10:49.099ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8282024-11-21T23:10:49.099ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62024
8292024-11-21T23:10:49.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7648 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62024 req_id = 8fbc985c-4594-44a5-94c5-6a8e6f775d67 response_code = 200 uri = /timesync
8302024-11-21T23:10:49.109ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8312024-11-21T23:10:49.109ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8322024-11-21T23:10:49.214ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 174.186421ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8332024-11-21T23:10:49.660ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8342024-11-21T23:10:49.660ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48928
8352024-11-21T23:10:49.668ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7939 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48928 req_id = 84a7cacf-c51e-4cee-b6e0-432b11a4f932 response_code = 200 uri = /timesync
8362024-11-21T23:10:49.670ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8372024-11-21T23:10:49.670ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8382024-11-21T23:10:49.711ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 329491034640178752 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8392024-11-21T23:10:49.711ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17995928283259856397 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8402024-11-21T23:10:50.052ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8412024-11-21T23:10:50.052ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46930
8422024-11-21T23:10:50.060ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8419 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46930 req_id = d2ab2c65-902f-4fde-bc53-87b2dad0fe76 response_code = 200 uri = /timesync
8432024-11-21T23:10:50.062ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8442024-11-21T23:10:50.062ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8452024-11-21T23:10:50.603ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8462024-11-21T23:10:50.603ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51984
8472024-11-21T23:10:50.612ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8574 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51984 req_id = c042cb54-aa8e-455d-a9c9-c466e581b1ed response_code = 200 uri = /timesync
8482024-11-21T23:10:50.614ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8492024-11-21T23:10:50.614ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8502024-11-21T23:10:50.649ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5398698369582487023 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8512024-11-21T23:10:50.814ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15858676633825570490 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8522024-11-21T23:10:50.817ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2459447259745793341 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8532024-11-21T23:10:50.949ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2919013623316299948 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8542024-11-21T23:10:50.949ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5126378412384038234 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8552024-11-21T23:10:51.256ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:33590
8562024-11-21T23:10:51.256ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 89 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:33590 req_id = e4b74605-12e0-4384-9e67-5038221badcd response_code = 200 uri = /baseboard
8572024-11-21T23:10:51.368ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8582024-11-21T23:10:51.368ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38936
8592024-11-21T23:10:51.377ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8347 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38936 req_id = b153d715-c3c5-4786-96d4-04dafdf56152 response_code = 200 uri = /timesync
8602024-11-21T23:10:51.379ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8612024-11-21T23:10:51.379ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8622024-11-21T23:10:51.857ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8632024-11-21T23:10:51.857ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40778
8642024-11-21T23:10:51.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7951 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40778 req_id = be7a0519-585a-478e-a4b8-bbbbaaecf5fd response_code = 200 uri = /timesync
8652024-11-21T23:10:51.868ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8662024-11-21T23:10:51.868ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8672024-11-21T23:10:52.306ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8682024-11-21T23:10:52.306ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8692024-11-21T23:10:52.806ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8702024-11-21T23:10:52.806ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64128
8712024-11-21T23:10:52.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8420 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64128 req_id = 2b46072f-012a-4652-949c-75a2d0ca4d18 response_code = 200 uri = /timesync
8722024-11-21T23:10:52.819ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8732024-11-21T23:10:52.819ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8742024-11-21T23:10:53.732ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8752024-11-21T23:10:53.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34240
8762024-11-21T23:10:53.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8338 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34240 req_id = 1233fd8e-5c30-49b4-a3d1-e31c739bbf59 response_code = 200 uri = /timesync
8772024-11-21T23:10:53.745ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8782024-11-21T23:10:53.745ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8792024-11-21T23:10:55.635ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8802024-11-21T23:10:55.635ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45934
8812024-11-21T23:10:55.643ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45934 req_id = a92329e0-b835-4545-bf2f-55382dcdd844 response_code = 200 uri = /timesync
8822024-11-21T23:10:55.646ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8832024-11-21T23:10:55.646ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8842024-11-21T23:10:57.073ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8852024-11-21T23:10:57.073ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37674
8862024-11-21T23:10:57.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9228 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37674 req_id = 8a0b64bb-72db-4042-a04b-954492606962 response_code = 200 uri = /timesync
8872024-11-21T23:10:57.087ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8882024-11-21T23:10:57.087ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8892024-11-21T23:10:58.226ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
8902024-11-21T23:10:58.226ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50092
8912024-11-21T23:10:58.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8350 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50092 req_id = cb10acf4-a09f-4db2-bef1-1e59541c2243 response_code = 200 uri = /timesync
8922024-11-21T23:10:58.237ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
8932024-11-21T23:10:58.237ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
8942024-11-21T23:10:59.713ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 329491034640178752 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8952024-11-21T23:10:59.714ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17995928283259856397 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8962024-11-21T23:11:00.651ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5398698369582487023 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8972024-11-21T23:11:00.816ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15858676633825570490 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8982024-11-21T23:11:00.817ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2459447259745793341 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
8992024-11-21T23:11:00.950ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2919013623316299948 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9002024-11-21T23:11:00.951ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5126378412384038234 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9012024-11-21T23:11:01.019ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9022024-11-21T23:11:01.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39891
9032024-11-21T23:11:01.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 7992 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39891 req_id = 53e2de96-fa4d-444d-8c0f-52ab9e0dd533 response_code = 200 uri = /timesync
9042024-11-21T23:11:01.030ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9052024-11-21T23:11:01.030ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9062024-11-21T23:11:02.309ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9072024-11-21T23:11:02.309ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9082024-11-21T23:11:03.582ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9092024-11-21T23:11:03.583ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54504
9102024-11-21T23:11:03.591ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8723 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54504 req_id = 60c45d60-e451-4ba7-8a47-1ec572653850 response_code = 200 uri = /timesync
9112024-11-21T23:11:03.594ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9122024-11-21T23:11:03.594ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9132024-11-21T23:11:04.423ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 323.798286ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9142024-11-21T23:11:07.295ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9152024-11-21T23:11:07.295ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50673
9162024-11-21T23:11:07.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8638 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50673 req_id = 0feb1384-8862-4187-95f0-4cfa331e96f6 response_code = 200 uri = /timesync
9172024-11-21T23:11:07.307ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9182024-11-21T23:11:07.307ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9192024-11-21T23:11:09.716ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 329491034640178752 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9202024-11-21T23:11:09.716ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17995928283259856397 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9212024-11-21T23:11:10.653ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5398698369582487023 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9222024-11-21T23:11:10.818ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15858676633825570490 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9232024-11-21T23:11:10.818ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2459447259745793341 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9242024-11-21T23:11:10.951ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 2919013623316299948 reason = Duration(60s) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9252024-11-21T23:11:10.952ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 5126378412384038234 reason = Duration(60s) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9262024-11-21T23:11:12.311ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9272024-11-21T23:11:12.311ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9282024-11-21T23:11:12.413ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9292024-11-21T23:11:12.413ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44568
9302024-11-21T23:11:12.421ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8616 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44568 req_id = ee7b1448-b3bb-46f4-a11e-03d25675c837 response_code = 200 uri = /timesync
9312024-11-21T23:11:12.424ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9322024-11-21T23:11:12.424ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9332024-11-21T23:11:16.325ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9342024-11-21T23:11:16.325ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61516
9352024-11-21T23:11:16.333ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8338 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61516 req_id = 335557ee-c7ff-442b-9bab-411e2da280ab response_code = 200 uri = /timesync
9362024-11-21T23:11:16.336ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9372024-11-21T23:11:16.336ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9382024-11-21T23:11:19.695ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9392024-11-21T23:11:19.696ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63610
9402024-11-21T23:11:19.704ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8031 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63610 req_id = f3d96d26-25c7-47f6-80b4-736c7302a91c response_code = 200 uri = /timesync
9412024-11-21T23:11:19.706ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9422024-11-21T23:11:19.706ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9432024-11-21T23:11:19.718ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 329491034640178752 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9442024-11-21T23:11:19.718ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17995928283259856397 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9452024-11-21T23:11:19.784ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 995.656341ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9462024-11-21T23:11:20.655ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5398698369582487023 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9472024-11-21T23:11:20.821ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15858676633825570490 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9482024-11-21T23:11:20.821ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2459447259745793341 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9492024-11-21T23:11:21.292ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:45019
9502024-11-21T23:11:21.292ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 24 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:45019 req_id = 93e68275-0733-460a-a318-85888f4b44d9 response_code = 200 uri = /baseboard
9512024-11-21T23:11:22.313ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9522024-11-21T23:11:22.313ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9532024-11-21T23:11:28.803ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9542024-11-21T23:11:28.803ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41994
9552024-11-21T23:11:28.813ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9071 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41994 req_id = dd6585d7-e31b-45b2-8025-d17becfc0cb1 response_code = 200 uri = /timesync
9562024-11-21T23:11:28.815ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9572024-11-21T23:11:28.815ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9582024-11-21T23:11:29.719ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 329491034640178752 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9592024-11-21T23:11:29.720ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17995928283259856397 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9602024-11-21T23:11:30.657ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 5398698369582487023 reason = Duration(60s) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9612024-11-21T23:11:30.822ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 15858676633825570490 reason = Duration(60s) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9622024-11-21T23:11:30.822ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 2459447259745793341 reason = Duration(60s) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9632024-11-21T23:11:32.315ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9642024-11-21T23:11:32.315ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9652024-11-21T23:11:34.792ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9662024-11-21T23:11:34.793ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50500
9672024-11-21T23:11:34.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8838 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50500 req_id = e4497ca4-3934-4838-85c3-9f6073e7b027 response_code = 200 uri = /timesync
9682024-11-21T23:11:34.804ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9692024-11-21T23:11:34.804ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9702024-11-21T23:11:35.815ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.759790623s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9712024-11-21T23:11:39.721ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 329491034640178752 reason = Duration(60s) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9722024-11-21T23:11:39.721ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 17995928283259856397 reason = Duration(60s) sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9732024-11-21T23:11:42.316ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9742024-11-21T23:11:42.316ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9752024-11-21T23:11:48.971ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9762024-11-21T23:11:48.971ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42642
9772024-11-21T23:11:48.981ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 9706 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42642 req_id = 8539606d-73c7-4e74-8f61-1a8b6809b4de response_code = 200 uri = /timesync
9782024-11-21T23:11:48.983ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9792024-11-21T23:11:48.983ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9802024-11-21T23:11:51.328ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:48480
9812024-11-21T23:11:51.328ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 33 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:48480 req_id = 873e47a6-8f54-4b8b-9462-d0241445c501 response_code = 200 uri = /baseboard
9822024-11-21T23:11:52.319ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9832024-11-21T23:11:52.319ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9842024-11-21T23:11:52.612ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 5.732934179s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9852024-11-21T23:11:58.242ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9862024-11-21T23:11:58.242ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36808
9872024-11-21T23:11:58.251ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8340 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36808 req_id = ea7fcac9-cc22-4540-bdde-31d0b847bffa response_code = 200 uri = /timesync
9882024-11-21T23:11:58.253ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9892024-11-21T23:11:58.253ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
9902024-11-21T23:12:02.320ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9912024-11-21T23:12:02.320ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9922024-11-21T23:12:12.322ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9932024-11-21T23:12:12.322ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9942024-11-21T23:12:13.380ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 10.839542407s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
9952024-11-21T23:12:16.745ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
9962024-11-21T23:12:16.745ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46943
9972024-11-21T23:12:16.754ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8682 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46943 req_id = 4e3e9de3-3581-42c1-8f7c-2866d88ce291 response_code = 200 uri = /timesync
9982024-11-21T23:12:16.756ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
9992024-11-21T23:12:16.756ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10002024-11-21T23:12:21.362ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:65429
10012024-11-21T23:12:21.362ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 31 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:65429 req_id = 300fccf3-dcee-4956-932c-046f55911921 response_code = 200 uri = /baseboard
10022024-11-21T23:12:22.325ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10032024-11-21T23:12:22.325ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10042024-11-21T23:12:32.327ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10052024-11-21T23:12:32.327ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10062024-11-21T23:12:34.218ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
10072024-11-21T23:12:34.218ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60318
10082024-11-21T23:12:34.227ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8833 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60318 req_id = a696ced5-7019-4812-83c2-97d5ccf2f18d response_code = 200 uri = /timesync
10092024-11-21T23:12:34.231ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
10102024-11-21T23:12:34.231ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10112024-11-21T23:12:39.256ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 10.025948118s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10122024-11-21T23:12:42.329ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10132024-11-21T23:12:42.330ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10142024-11-21T23:12:51.395ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:63229
10152024-11-21T23:12:51.395ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 36 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:63229 req_id = 7a602bb5-caaf-4f9d-a7ec-370eb0f4dcf9 response_code = 200 uri = /baseboard
10162024-11-21T23:12:52.331ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10172024-11-21T23:12:52.331ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10182024-11-21T23:12:58.675ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
10192024-11-21T23:12:58.675ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60367
10202024-11-21T23:12:58.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8611 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60367 req_id = 29c5efcf-003d-4d11-9e09-b69f05f869ce response_code = 200 uri = /timesync
10212024-11-21T23:12:58.686ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
10222024-11-21T23:12:58.686ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10232024-11-21T23:13:02.333ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10242024-11-21T23:13:02.333ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10252024-11-21T23:13:04.318ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 33.610211075s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10262024-11-21T23:13:12.335ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10272024-11-21T23:13:12.335ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10282024-11-21T23:13:13.100ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
10292024-11-21T23:13:13.100ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57488
10302024-11-21T23:13:13.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8759 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57488 req_id = e340bfbb-00a1-47fe-89dd-25140671e964 response_code = 200 uri = /timesync
10312024-11-21T23:13:13.113ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
10322024-11-21T23:13:13.113ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10332024-11-21T23:13:21.428ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:33084
10342024-11-21T23:13:21.429ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 36 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:33084 req_id = 7b4c5c38-9b7d-4989-b8dc-43a296f08151 response_code = 200 uri = /baseboard
10352024-11-21T23:13:22.338ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10362024-11-21T23:13:22.338ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10372024-11-21T23:13:32.340ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10382024-11-21T23:13:32.340ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10392024-11-21T23:13:35.217ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
10402024-11-21T23:13:35.218ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62103
10412024-11-21T23:13:35.226ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8433 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62103 req_id = 62627fdf-bf02-4d93-b284-d274046c053a response_code = 200 uri = /timesync
10422024-11-21T23:13:35.228ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
10432024-11-21T23:13:35.228ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10442024-11-21T23:13:42.343ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10452024-11-21T23:13:42.343ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10462024-11-21T23:13:51.460ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:41757
10472024-11-21T23:13:51.460ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 43 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:41757 req_id = 82b1c7e4-535c-4e14-9da4-b24dfca6378b response_code = 200 uri = /baseboard
10482024-11-21T23:13:52.344ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10492024-11-21T23:13:52.345ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10502024-11-21T23:13:52.963ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 61.106793434s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10512024-11-21T23:14:02.346ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10522024-11-21T23:14:02.346ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10532024-11-21T23:14:12.348ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10542024-11-21T23:14:12.348ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10552024-11-21T23:14:14.097ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
10562024-11-21T23:14:14.097ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59970
10572024-11-21T23:14:14.105ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8318 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59970 req_id = 69a34c80-554b-430a-9073-ef7f981125b3 response_code = 200 uri = /timesync
10582024-11-21T23:14:14.109ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
10592024-11-21T23:14:14.109ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10602024-11-21T23:14:21.495ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:40075
10612024-11-21T23:14:21.495ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 27 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:40075 req_id = 9f05c37c-5f12-4db0-bd49-6d9c09c9bf5d response_code = 200 uri = /baseboard
10622024-11-21T23:14:22.350ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10632024-11-21T23:14:22.350ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10642024-11-21T23:14:32.353ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10652024-11-21T23:14:32.353ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10662024-11-21T23:14:42.355ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10672024-11-21T23:14:42.356ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10682024-11-21T23:14:47.858ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
10692024-11-21T23:14:47.858ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55253
10702024-11-21T23:14:47.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8229 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55253 req_id = 3c7634c8-05bf-4613-9710-f2c8a798ff32 response_code = 200 uri = /timesync
10712024-11-21T23:14:47.870ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
10722024-11-21T23:14:47.870ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10732024-11-21T23:14:51.528ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:37568
10742024-11-21T23:14:51.528ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 26 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:37568 req_id = 08fa8e36-d002-4730-b4f2-4e0545b86236 response_code = 200 uri = /baseboard
10752024-11-21T23:14:52.358ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10762024-11-21T23:14:52.358ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10772024-11-21T23:15:02.359ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10782024-11-21T23:15:02.359ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10792024-11-21T23:15:09.107ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 187.159996811s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10802024-11-21T23:15:12.361ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10812024-11-21T23:15:12.361ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10822024-11-21T23:15:15.251ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
10832024-11-21T23:15:21.563ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:50806
10842024-11-21T23:15:21.563ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 36 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:50806 req_id = 3ebcaf5f-640e-42f5-b9c1-512308b8a3a3 response_code = 200 uri = /baseboard
10852024-11-21T23:15:22.363ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10862024-11-21T23:15:22.364ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10872024-11-21T23:15:29.021ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
10882024-11-21T23:15:29.021ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45886
10892024-11-21T23:15:29.030ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8878 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45886 req_id = fcc61e0a-7afb-4e11-9176-a9806732299c response_code = 200 uri = /timesync
10902024-11-21T23:15:29.033ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
10912024-11-21T23:15:29.033ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
10922024-11-21T23:15:32.365ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10932024-11-21T23:15:32.365ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10942024-11-21T23:15:42.367ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10952024-11-21T23:15:42.367ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10962024-11-21T23:15:51.597ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:42923
10972024-11-21T23:15:51.597ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 27 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:42923 req_id = c424417d-28eb-46fa-8003-30ced83f7b93 response_code = 200 uri = /baseboard
10982024-11-21T23:15:52.369ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
10992024-11-21T23:15:52.369ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11002024-11-21T23:16:02.371ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11012024-11-21T23:16:02.371ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11022024-11-21T23:16:12.373ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11032024-11-21T23:16:12.373ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11042024-11-21T23:16:21.631ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:53615
11052024-11-21T23:16:21.631ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 28 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:53615 req_id = f73103ef-0d3b-4e66-a813-48f1ea381ae0 response_code = 200 uri = /baseboard
11062024-11-21T23:16:22.376ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11072024-11-21T23:16:22.376ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11082024-11-21T23:16:32.378ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11092024-11-21T23:16:32.378ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11102024-11-21T23:16:42.380ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11112024-11-21T23:16:42.380ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11122024-11-21T23:16:43.829ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
11132024-11-21T23:16:43.829ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36021
11142024-11-21T23:16:43.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8523 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36021 req_id = 096bf963-10c4-47c8-ab92-2334640d99be response_code = 200 uri = /timesync
11152024-11-21T23:16:43.841ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
11162024-11-21T23:16:43.841ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
11172024-11-21T23:16:51.667ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:40546
11182024-11-21T23:16:51.667ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 45 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:40546 req_id = 979ff9e5-bafb-42af-b346-41b49c511c28 response_code = 200 uri = /baseboard
11192024-11-21T23:16:52.382ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11202024-11-21T23:16:52.383ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11212024-11-21T23:17:02.385ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11222024-11-21T23:17:02.385ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11232024-11-21T23:17:12.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11242024-11-21T23:17:12.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11252024-11-21T23:17:21.701ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:41875
11262024-11-21T23:17:21.701ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 37 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:41875 req_id = e8b640b3-3f14-480a-be27-fe4798721d43 response_code = 200 uri = /baseboard
11272024-11-21T23:17:22.389ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11282024-11-21T23:17:22.389ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11292024-11-21T23:17:27.761ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
11302024-11-21T23:17:27.761ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59235
11312024-11-21T23:17:27.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 8535 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59235 req_id = a2dea341-7057-4a73-a181-f5b02e62b278 response_code = 200 uri = /timesync
11322024-11-21T23:17:27.773ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:740
11332024-11-21T23:17:27.773ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
11342024-11-21T23:17:32.391ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11352024-11-21T23:17:32.391ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11362024-11-21T23:17:42.393ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11372024-11-21T23:17:42.394ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11382024-11-21T23:17:51.737ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:52366
11392024-11-21T23:17:51.737ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 46 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:52366 req_id = e89b4897-a644-4967-bf99-d9669191c408 response_code = 200 uri = /baseboard
11402024-11-21T23:17:52.395ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11412024-11-21T23:17:52.396ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11422024-11-21T23:18:02.398ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11432024-11-21T23:18:02.398ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11442024-11-21T23:18:12.401ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11452024-11-21T23:18:12.401ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11462024-11-21T23:18:21.769ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:61779
11472024-11-21T23:18:21.769ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 28 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:61779 req_id = 4980c812-e597-4fc1-a833-5ea20bb84d16 response_code = 200 uri = /baseboard
11482024-11-21T23:18:22.403ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11492024-11-21T23:18:22.403ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11502024-11-21T23:18:31.303ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 177.567589341s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:420 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11512024-11-21T23:18:32.404ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11522024-11-21T23:18:32.405ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11532024-11-21T23:18:42.406ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11542024-11-21T23:18:42.407ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11552024-11-21T23:18:51.803ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:57586
11562024-11-21T23:18:51.803ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 25 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:57586 req_id = 09d681ce-839c-44bc-b2d1-2ef71f847cbc response_code = 200 uri = /baseboard
11572024-11-21T23:18:52.408ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11582024-11-21T23:18:52.408ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11592024-11-21T23:18:59.446ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:712
11602024-11-21T23:18:59.446ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38614
11612024-11-21T23:18:59.455ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_71e73077-bc8e-4f9f-8689-878cda8ba43e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3958
11622024-11-21T23:18:59.455ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:851 latency_us = 8848 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38614 req_id = 22c729bf-bb47-4833-a435-4018979f9e51 response_code = 500 uri = /timesync
11632024-11-21T23:18:59.459ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:762
11642024-11-21T23:19:02.411ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11652024-11-21T23:19:02.411ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11662024-11-21T23:19:12.412ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11672024-11-21T23:19:12.412ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11682024-11-21T23:19:21.837ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:50876
11692024-11-21T23:19:21.837ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 34 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:50876 req_id = 65db33d1-c795-4f8c-a695-f90e490d770b response_code = 200 uri = /baseboard
11702024-11-21T23:19:22.415ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9374031121665754717 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534
11712024-11-21T23:19:22.415ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12244715441543845500 sled_id = df02b17d-a4bd-4ac6-9447-5abe2628a534