Max level shown:
1 [ Sep 7 18:50:31 Enabled. ]
2 [ Sep 7 18:50:31 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Sep 7 18:50:31 Method "start" exited with status 0. ]
4 [ Sep 7 18:50:31 Rereading configuration. ]
5 [ Sep 7 18:50:31 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-09-07T18:50:31.369ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-09-07T18:50:31.369ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-09-07T18:50:31.421ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-09-07T18:50:31.421ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-09-07T18:50:31.533ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-09-07T18:50:31.542ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-09-07T18:50:31.550ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-09-07T18:50:32.152ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:200::, width: 64 }
152024-09-07T18:50:32.288ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-09-07T18:50:32.288ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-09-07T18:50:32.288ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-09-07T18:50:32.288ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-09-07T18:50:32.288ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-09-07T18:50:32.288ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-09-07T18:50:32.291ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-09-07T18:50:32.291ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-09-07T18:50:32.291ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-09-07T18:50:32.291ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-09-07T18:50:32.291ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-09-07T18:50:32.292ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-09-07T18:50:32.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-09-07T18:50:32.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-09-07T18:50:32.292ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-09-07T18:50:32.303ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-09-07T18:50:32.303ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool)
322024-09-07T18:50:32.403ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal }
332024-09-07T18:50:32.814ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-09-07T18:50:32.814ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-09-07T18:50:32.814ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-09-07T18:50:32.814ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-09-07T18:50:32.814ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-09-07T18:50:32.814ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-09-07T18:50:32.814ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-09-07T18:50:32.823ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-09-07T18:50:32.823ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = f677c761-8af0-4ab3-b20f-2436c20df40c (zpool)
422024-09-07T18:50:32.909ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal }
432024-09-07T18:50:33.307ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-09-07T18:50:33.307ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-09-07T18:50:33.307ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-09-07T18:50:33.307ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-09-07T18:50:33.307ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-09-07T18:50:33.307ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-09-07T18:50:33.307ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-09-07T18:50:33.307ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-09-07T18:50:33.307ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
522024-09-07T18:50:33.307ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
532024-09-07T18:50:33.307ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
542024-09-07T18:50:33.307ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-09-07T18:50:33.307ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-09-07T18:50:33.307ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
602024-09-07T18:50:33.308ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-09-07T18:50:33.308ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-09-07T18:50:33.308ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
642024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
652024-09-07T18:50:33.308ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
662024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-09-07T18:50:33.308ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-09-07T18:50:33.308ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-09-07T18:50:33.308ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-09-07T18:50:33.308ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-09-07T18:50:33.308ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
782024-09-07T18:50:33.308ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-09-07T18:50:33.308ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
812024-09-07T18:50:33.308ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
822024-09-07T18:50:33.308ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-09-07T18:50:33.308ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-09-07T18:50:33.308ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-09-07T18:50:33.308ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-09-07T18:50:33.308ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-09-07T18:50:33.308ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
942024-09-07T18:50:33.308ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
952024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-09-07T18:50:33.308ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-09-07T18:50:33.308ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-09-07T18:50:33.308ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-09-07T18:50:33.308ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-09-07T18:50:33.308ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1032024-09-07T18:50:33.308ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-09-07T18:50:33.308ZINFOSledAgent (bootstore): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-09-07T18:50:33.308ZINFOSledAgent (bootstore): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-09-07T18:50:33.308ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-09-07T18:50:33.308ZINFOSledAgent (bootstore): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-09-07T18:50:33.308ZINFOSledAgent (bootstore): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-09-07T18:50:33.308ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-09-07T18:50:33.308ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-09-07T18:50:33.308ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-09-07T18:50:33.318ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:730
1132024-09-07T18:50:33.318ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1083, tv_nsec: 967839976 } time_to_next_cleanup = 599.999946489s
1142024-09-07T18:50:33.319ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3842
1152024-09-07T18:50:33.319ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-09-07T18:50:33.319ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-09-07T18:50:33.319ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-09-07T18:50:33.322ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:200::1]:80
1192024-09-07T18:50:33.322ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:200::1]:12346
1202024-09-07T18:50:33.322ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-09-07T18:50:33.322ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1222024-09-07T18:50:33.322ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-09-07T18:50:33.322ZINFOSledAgent (RSS): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-09-07T18:50:33.322ZINFOSledAgent (RSS): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-09-07T18:50:33.322ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-09-07T18:50:33.322ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-09-07T18:50:33.322ZINFOSledAgent (RSS): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-09-07T18:50:33.322ZINFOSledAgent (RSS): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-09-07T18:50:33.322ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-09-07T18:50:33.322ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1312024-09-07T18:50:33.322ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:200::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1322024-09-07T18:50:33.322ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-09-07T18:50:33.322ZINFOSledAgent (RSS): Writing ledger to /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-09-07T18:50:33.323ZINFOSledAgent (RSS): Writing ledger to /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-09-07T18:50:33.323ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 890b6229-64f5-47ca-9c4f-4917b1f665e3, sleds: { [fdb0:200::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: da86b797-b25a-4724-8ccb-680ac8134295, rack_id: 890b6229-64f5-47ca-9c4f-4917b1f665e3, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1362024-09-07T18:50:33.323ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-09-07T18:50:33.323ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-09-07T18:50:33.323ZINFOSledAgent (bootstore): Writing ledger to /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-09-07T18:50:33.323ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-09-07T18:50:33.323ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-09-07T18:50:33.323ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:221 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: da86b797-b25a-4724-8ccb-680ac8134295, rack_id: 890b6229-64f5-47ca-9c4f-4917b1f665e3, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:200::1]:12346
1422024-09-07T18:50:33.324ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:200::1]:63533
1432024-09-07T18:50:33.324ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: da86b797-b25a-4724-8ccb-680ac8134295, rack_id: 890b6229-64f5-47ca-9c4f-4917b1f665e3, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-09-07T18:50:33.324ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1462024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1472024-09-07T18:50:33.324ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1482024-09-07T18:50:33.324ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1492024-09-07T18:50:33.324ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1502024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1512024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1522024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1532024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1542024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1552024-09-07T18:50:33.324ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1562024-09-07T18:50:33.324ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1572024-09-07T18:50:33.326ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4139
1582024-09-07T18:50:33.326ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4521
1592024-09-07T18:50:33.358ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1602024-09-07T18:50:33.380ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-09-07T18:50:33.397ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-09-07T18:50:33.408ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:388 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1632024-09-07T18:50:33.408ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1642024-09-07T18:50:33.408ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:400 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1652024-09-07T18:50:33.408ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-09-07T18:50:33.411ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 93.418275ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "98e2e7c0-076b-4f18-9c6b-daf725309653", "content-length": "124", "date": "Sat, 07 Sep 2024 18:50:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "98e2e7c0-076b-4f18-9c6b-daf725309653" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-09-07T18:50:33.421ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_951f74f7-28ed-48ce-8b8e-a82d754c6f9c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-09-07T18:50:33.445ZINFOSledAgent: successfully created encrypted zvol: name="oxi_951f74f7-28ed-48ce-8b8e-a82d754c6f9c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-09-07T18:50:33.445ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_951f74f7-28ed-48ce-8b8e-a82d754c6f9c/swap" file = sled-agent/src/swap_device.rs:96
1702024-09-07T18:50:33.450ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:415 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1712024-09-07T18:50:33.450ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:127
1722024-09-07T18:50:33.450ZINFOSledAgent (BackingFs): Ensuring dataset oxi_951f74f7-28ed-48ce-8b8e-a82d754c6f9c/backing/fmd file = sled-agent/src/backing_fs.rs:137
1732024-09-07T18:50:33.506ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-09-07T18:50:33.506ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 75.869637ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "080c8b10-8ff2-4a8a-bedf-4d9144bf3d26", "content-length": "124", "date": "Sat, 07 Sep 2024 18:50:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "080c8b10-8ff2-4a8a-bedf-4d9144bf3d26" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-09-07T18:50:33.537ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:172
1762024-09-07T18:50:33.584ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-09-07T18:50:33.584ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 110.108494ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "32fe777a-a94e-4cf6-8151-4c4e8066c261", "content-length": "124", "date": "Sat, 07 Sep 2024 18:50:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "32fe777a-a94e-4cf6-8151-4c4e8066c261" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-09-07T18:50:33.695ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-09-07T18:50:33.696ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 201.176489ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e0e77153-5ab2-4e3e-95c9-d3869bde565d", "content-length": "124", "date": "Sat, 07 Sep 2024 18:50:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e0e77153-5ab2-4e3e-95c9-d3869bde565d" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-09-07T18:50:33.756ZINFOSledAgent (BackingFs): Mounting oxi_951f74f7-28ed-48ce-8b8e-a82d754c6f9c/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:180
1812024-09-07T18:50:33.769ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:188
1822024-09-07T18:50:33.769ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:188
1832024-09-07T18:50:33.769ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:188
1842024-09-07T18:50:33.769ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:194
1852024-09-07T18:50:33.898ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-09-07T18:50:33.899ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 248.169236ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "933f0855-0987-40f6-81ba-360f3edb5f68", "content-length": "124", "date": "Sat, 07 Sep 2024 18:50:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "933f0855-0987-40f6-81ba-360f3edb5f68" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-09-07T18:50:34.148ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-09-07T18:50:34.149ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 231.894002ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6b797c21-f3a0-4843-ba50-c114a9bada3e", "content-length": "124", "date": "Sat, 07 Sep 2024 18:50:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6b797c21-f3a0-4843-ba50-c114a9bada3e" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-09-07T18:50:34.381ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1902024-09-07T18:50:34.382ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 367.68456ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2bc258db-5d03-4b56-969d-081e685dcb47", "content-length": "124", "date": "Sat, 07 Sep 2024 18:50:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2bc258db-5d03-4b56-969d-081e685dcb47" } file = clients/ddm-admin-client/src/lib.rs:177
1912024-09-07T18:50:34.505ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1922024-09-07T18:50:34.510ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:47084 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1932024-09-07T18:50:34.510ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1942024-09-07T18:50:34.510ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:47084 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = da86b797-b25a-4724-8ccb-680ac8134295 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1952024-09-07T18:50:34.511ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 273.274988ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1962024-09-07T18:50:34.512ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1972024-09-07T18:50:34.512ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
1982024-09-07T18:50:34.512ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:914 underlay_address = fd00:1122:3344:101::1
1992024-09-07T18:50:34.512ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
2002024-09-07T18:50:34.750ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2012024-09-07T18:50:34.787ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 402.433401ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2022024-09-07T18:50:35.192ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 648.791176ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2032024-09-07T18:50:35.842ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.9619126s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2042024-09-07T18:50:36.949ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2615
2052024-09-07T18:50:36.949ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2833
2062024-09-07T18:50:36.949ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2485
2072024-09-07T18:50:36.949ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2532
2082024-09-07T18:50:36.949ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2914
2092024-09-07T18:50:36.949ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2956
2102024-09-07T18:50:36.951ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2758
2112024-09-07T18:50:36.952ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2526
2122024-09-07T18:50:36.952ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='d2b38bdc-5d2d-469e-9721-989d9af341b1'/> <propval type="astring" name="rack_id" value='890b6229-64f5-47ca-9c4f-4917b1f665e3'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:200::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fed7:e443'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:200::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='da86b797-b25a-4724-8ccb-680ac8134295'/> <propval type="astring" name="rack_id" value='890b6229-64f5-47ca-9c4f-4917b1f665e3'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='da86b797-b25a-4724-8ccb-680ac8134295'/> <propval type="astring" name="rack_uuid" value='890b6229-64f5-47ca-9c4f-4917b1f665e3'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='da86b797-b25a-4724-8ccb-680ac8134295'/> <propval type="astring" name="rack_uuid" value='890b6229-64f5-47ca-9c4f-4917b1f665e3'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2132024-09-07T18:50:36.952ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2142024-09-07T18:50:37.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 62.40124ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2152024-09-07T18:50:37.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 87.913897ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2162024-09-07T18:50:37.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 200.220641ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2172024-09-07T18:50:37.807ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.75658395s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2182024-09-07T18:50:38.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 471.537279ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-09-07T18:50:38.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.170551388s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-09-07T18:50:39.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.155110096s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-09-07T18:50:41.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.123504259s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-09-07T18:50:42.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 966.24416ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-09-07T18:50:42.572ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 7.240899202s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2242024-09-07T18:50:43.625ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2252024-09-07T18:50:43.625ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2262024-09-07T18:50:43.625ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2272024-09-07T18:50:43.625ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2282024-09-07T18:50:43.626ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2292024-09-07T18:50:43.626ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2302024-09-07T18:50:43.626ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:200::1]:63533
2312024-09-07T18:50:43.626ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2322024-09-07T18:50:43.626ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3842
2332024-09-07T18:50:43.626ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:112 peer = [fdb0:200::1]:12346
2342024-09-07T18:50:43.839ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:236 target_sled = [fdb0:200::1]:12346
2352024-09-07T18:50:43.839ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2362024-09-07T18:50:43.839ZINFOSledAgent (RSS): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2372024-09-07T18:50:43.839ZINFOSledAgent (RSS): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2382024-09-07T18:50:43.839ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2392024-09-07T18:50:43.839ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2402024-09-07T18:50:43.839ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-09-07T18:50:43.844ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4163 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2422024-09-07T18:50:43.844ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4179
2432024-09-07T18:50:43.844ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2442024-09-07T18:50:44.111ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36747
2452024-09-07T18:50:44.112ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-09-07T18:50:44.112ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 341 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36747 req_id = 0c0a0a87-ed7b-41bf-939a-1e430ceea3c3 response_code = 200 uri = /inventory
2472024-09-07T18:50:44.143ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38074
2482024-09-07T18:50:44.143ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38074 req_id = 881ec2bf-ea70-4a52-8183-0204b3ed0e5c response_code = 200 uri = /sled-role
2492024-09-07T18:50:44.145ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2502024-09-07T18:50:44.145ZINFOSledAgent (RSS): Writing ledger to /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2512024-09-07T18:50:44.145ZINFOSledAgent (RSS): Writing ledger to /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2522024-09-07T18:50:44.145ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:1001
2532024-09-07T18:50:44.178ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 5882cd91-c8a9-45db-b4ad-9a7289fe66a2, pool_id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: ef8b359f-f47c-4ed0-9bd5-ab7a7010b669, pool_id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 91a10670-917c-4847-88af-6c08d48cec51, pool_id: faa0cc11-3e46-4294-868c-12831b857884 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 143fecea-ef54-4258-879f-96832bc6cbd5, pool_id: f5b9b071-b292-42ff-8f22-90770e6df42a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 614cc534-1c05-452f-826e-78d16b6aa53e, pool_id: 0b0f060f-15eb-447f-a739-7bce57c42c4b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 72da3f71-9fc0-4238-86ce-427fef2fda93, pool_id: 9d00b4ab-4f70-4f02-8fff-824dda813e40 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 34cad5af-ab67-4b75-8194-572995587f77, pool_id: 78ea79f6-adbf-4571-a535-37760fc84bd0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: aec18675-7f24-4c08-8823-2f61d9b2a33a, pool_id: cb8c5357-055f-48a8-ad91-4c8525191c8d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 40c54004-0c0b-446d-aca4-8c448ee9ceed, pool_id: 50801f60-af2f-4f08-8e46-106a71ec966e (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2542024-09-07T18:50:44.178ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53635
2552024-09-07T18:50:44.178ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:856 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2562024-09-07T18:50:44.178ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 5882cd91-c8a9-45db-b4ad-9a7289fe66a2, pool_id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: ef8b359f-f47c-4ed0-9bd5-ab7a7010b669, pool_id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 91a10670-917c-4847-88af-6c08d48cec51, pool_id: faa0cc11-3e46-4294-868c-12831b857884 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 143fecea-ef54-4258-879f-96832bc6cbd5, pool_id: f5b9b071-b292-42ff-8f22-90770e6df42a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 614cc534-1c05-452f-826e-78d16b6aa53e, pool_id: 0b0f060f-15eb-447f-a739-7bce57c42c4b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 72da3f71-9fc0-4238-86ce-427fef2fda93, pool_id: 9d00b4ab-4f70-4f02-8fff-824dda813e40 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 34cad5af-ab67-4b75-8194-572995587f77, pool_id: 78ea79f6-adbf-4571-a535-37760fc84bd0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: aec18675-7f24-4c08-8823-2f61d9b2a33a, pool_id: cb8c5357-055f-48a8-ad91-4c8525191c8d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 40c54004-0c0b-446d-aca4-8c448ee9ceed, pool_id: 50801f60-af2f-4f08-8e46-106a71ec966e (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2572024-09-07T18:50:44.178ZINFOSledAgent (StorageManager): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2582024-09-07T18:50:44.178ZINFOSledAgent (StorageManager): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2592024-09-07T18:50:44.178ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2602024-09-07T18:50:44.178ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2612024-09-07T18:50:44.178ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2622024-09-07T18:50:44.178ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2632024-09-07T18:50:44.178ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2642024-09-07T18:50:44.317ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2652024-09-07T18:50:44.317ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool)
2662024-09-07T18:50:44.570ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2672024-09-07T18:50:44.628ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/da86b797-b25a-4724-8ccb-680ac8134295): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
2682024-09-07T18:50:44.664ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }
2692024-09-07T18:50:44.672ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:218
2702024-09-07T18:50:44.672ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:220
2712024-09-07T18:50:44.672ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:224
2722024-09-07T18:50:44.672ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:226
2732024-09-07T18:50:44.672ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2742024-09-07T18:50:44.672ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2752024-09-07T18:50:44.672ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
2762024-09-07T18:50:44.709ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2772024-09-07T18:50:44.717ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_308c8624-6854-4c09-8fdb-403d3e7020c4/crypt/zone file = sled-storage/src/dataset.rs:281
2782024-09-07T18:50:44.893ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_308c8624-6854-4c09-8fdb-403d3e7020c4 file = sled-storage/src/dataset.rs:367
2792024-09-07T18:50:44.901ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2802024-09-07T18:50:44.901ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2812024-09-07T18:50:44.901ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2822024-09-07T18:50:44.901ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2832024-09-07T18:50:44.901ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2842024-09-07T18:50:44.911ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2852024-09-07T18:50:44.911ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool)
2862024-09-07T18:50:45.004ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External }
2872024-09-07T18:50:45.011ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:218
2882024-09-07T18:50:45.011ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:220
2892024-09-07T18:50:45.011ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:224
2902024-09-07T18:50:45.011ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:226
2912024-09-07T18:50:45.012ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2922024-09-07T18:50:45.012ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2932024-09-07T18:50:45.012ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
2942024-09-07T18:50:45.051ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2952024-09-07T18:50:45.060ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7c49d716-0b82-4096-a83b-a43dd8cb57a5/crypt/zone file = sled-storage/src/dataset.rs:281
2962024-09-07T18:50:45.229ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7c49d716-0b82-4096-a83b-a43dd8cb57a5 file = sled-storage/src/dataset.rs:367
2972024-09-07T18:50:45.237ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
2982024-09-07T18:50:45.237ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
2992024-09-07T18:50:45.237ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3002024-09-07T18:50:45.237ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3012024-09-07T18:50:45.237ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3022024-09-07T18:50:45.246ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3032024-09-07T18:50:45.246ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = faa0cc11-3e46-4294-868c-12831b857884 (zpool)
3042024-09-07T18:50:45.337ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External }
3052024-09-07T18:50:45.345ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:218
3062024-09-07T18:50:45.345ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:220
3072024-09-07T18:50:45.345ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:224
3082024-09-07T18:50:45.345ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:226
3092024-09-07T18:50:45.345ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3102024-09-07T18:50:45.345ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3112024-09-07T18:50:45.345ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3122024-09-07T18:50:45.381ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3132024-09-07T18:50:45.389ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_faa0cc11-3e46-4294-868c-12831b857884/crypt/zone file = sled-storage/src/dataset.rs:281
3142024-09-07T18:50:45.564ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_faa0cc11-3e46-4294-868c-12831b857884 file = sled-storage/src/dataset.rs:367
3152024-09-07T18:50:45.572ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3162024-09-07T18:50:45.572ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3172024-09-07T18:50:45.572ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3182024-09-07T18:50:45.572ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3192024-09-07T18:50:45.572ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3202024-09-07T18:50:45.581ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3212024-09-07T18:50:45.581ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = f5b9b071-b292-42ff-8f22-90770e6df42a (zpool)
3222024-09-07T18:50:45.674ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: f5b9b071-b292-42ff-8f22-90770e6df42a (zpool), kind: External }
3232024-09-07T18:50:45.682ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:218
3242024-09-07T18:50:45.682ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:220
3252024-09-07T18:50:45.682ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:224
3262024-09-07T18:50:45.682ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:226
3272024-09-07T18:50:45.682ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3282024-09-07T18:50:45.682ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3292024-09-07T18:50:45.682ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3302024-09-07T18:50:45.686ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/da86b797-b25a-4724-8ccb-680ac8134295): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
3312024-09-07T18:50:45.719ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3322024-09-07T18:50:45.728ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f5b9b071-b292-42ff-8f22-90770e6df42a/crypt/zone file = sled-storage/src/dataset.rs:281
3332024-09-07T18:50:45.901ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f5b9b071-b292-42ff-8f22-90770e6df42a file = sled-storage/src/dataset.rs:367
3342024-09-07T18:50:45.910ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3352024-09-07T18:50:45.910ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3362024-09-07T18:50:45.910ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3372024-09-07T18:50:45.910ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3382024-09-07T18:50:45.910ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3392024-09-07T18:50:45.920ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3402024-09-07T18:50:45.920ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 0b0f060f-15eb-447f-a739-7bce57c42c4b (zpool)
3412024-09-07T18:50:46.015ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 0b0f060f-15eb-447f-a739-7bce57c42c4b (zpool), kind: External }
3422024-09-07T18:50:46.024ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:218
3432024-09-07T18:50:46.024ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:220
3442024-09-07T18:50:46.024ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:224
3452024-09-07T18:50:46.024ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:226
3462024-09-07T18:50:46.024ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3472024-09-07T18:50:46.024ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3482024-09-07T18:50:46.024ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3492024-09-07T18:50:46.061ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3502024-09-07T18:50:46.069ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_0b0f060f-15eb-447f-a739-7bce57c42c4b/crypt/zone file = sled-storage/src/dataset.rs:281
3512024-09-07T18:50:46.247ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_0b0f060f-15eb-447f-a739-7bce57c42c4b file = sled-storage/src/dataset.rs:367
3522024-09-07T18:50:46.256ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3532024-09-07T18:50:46.256ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3542024-09-07T18:50:46.256ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3552024-09-07T18:50:46.256ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3562024-09-07T18:50:46.256ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3572024-09-07T18:50:46.265ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3582024-09-07T18:50:46.265ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 9d00b4ab-4f70-4f02-8fff-824dda813e40 (zpool)
3592024-09-07T18:50:46.359ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 9d00b4ab-4f70-4f02-8fff-824dda813e40 (zpool), kind: External }
3602024-09-07T18:50:46.366ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:218
3612024-09-07T18:50:46.366ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:220
3622024-09-07T18:50:46.366ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:224
3632024-09-07T18:50:46.366ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:226
3642024-09-07T18:50:46.366ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3652024-09-07T18:50:46.367ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-09-07T18:50:46.367ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3672024-09-07T18:50:46.403ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3682024-09-07T18:50:46.411ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9d00b4ab-4f70-4f02-8fff-824dda813e40/crypt/zone file = sled-storage/src/dataset.rs:281
3692024-09-07T18:50:46.589ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9d00b4ab-4f70-4f02-8fff-824dda813e40 file = sled-storage/src/dataset.rs:367
3702024-09-07T18:50:46.597ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3712024-09-07T18:50:46.597ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3722024-09-07T18:50:46.597ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3732024-09-07T18:50:46.597ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3742024-09-07T18:50:46.597ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3752024-09-07T18:50:46.607ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3762024-09-07T18:50:46.607ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 78ea79f6-adbf-4571-a535-37760fc84bd0 (zpool)
3772024-09-07T18:50:46.700ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 78ea79f6-adbf-4571-a535-37760fc84bd0 (zpool), kind: External }
3782024-09-07T18:50:46.708ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:218
3792024-09-07T18:50:46.708ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:220
3802024-09-07T18:50:46.708ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:224
3812024-09-07T18:50:46.708ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:226
3822024-09-07T18:50:46.708ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3832024-09-07T18:50:46.708ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3842024-09-07T18:50:46.708ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3852024-09-07T18:50:46.713ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/da86b797-b25a-4724-8ccb-680ac8134295): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
3862024-09-07T18:50:46.745ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3872024-09-07T18:50:46.753ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_78ea79f6-adbf-4571-a535-37760fc84bd0/crypt/zone file = sled-storage/src/dataset.rs:281
3882024-09-07T18:50:46.934ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_78ea79f6-adbf-4571-a535-37760fc84bd0 file = sled-storage/src/dataset.rs:367
3892024-09-07T18:50:46.942ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3902024-09-07T18:50:46.942ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3912024-09-07T18:50:46.942ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3922024-09-07T18:50:46.942ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3932024-09-07T18:50:46.942ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3942024-09-07T18:50:46.951ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3952024-09-07T18:50:46.951ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = cb8c5357-055f-48a8-ad91-4c8525191c8d (zpool)
3962024-09-07T18:50:47.043ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: cb8c5357-055f-48a8-ad91-4c8525191c8d (zpool), kind: External }
3972024-09-07T18:50:47.050ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:218
3982024-09-07T18:50:47.050ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:220
3992024-09-07T18:50:47.050ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:224
4002024-09-07T18:50:47.050ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:226
4012024-09-07T18:50:47.050ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4022024-09-07T18:50:47.050ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4032024-09-07T18:50:47.050ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
4042024-09-07T18:50:47.086ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4052024-09-07T18:50:47.094ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cb8c5357-055f-48a8-ad91-4c8525191c8d/crypt/zone file = sled-storage/src/dataset.rs:281
4062024-09-07T18:50:47.261ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cb8c5357-055f-48a8-ad91-4c8525191c8d file = sled-storage/src/dataset.rs:367
4072024-09-07T18:50:47.269ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4082024-09-07T18:50:47.269ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4092024-09-07T18:50:47.269ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4102024-09-07T18:50:47.269ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4112024-09-07T18:50:47.269ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4122024-09-07T18:50:47.280ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4132024-09-07T18:50:47.280ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 50801f60-af2f-4f08-8e46-106a71ec966e (zpool)
4142024-09-07T18:50:47.393ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 50801f60-af2f-4f08-8e46-106a71ec966e (zpool), kind: External }
4152024-09-07T18:50:47.400ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:218
4162024-09-07T18:50:47.400ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:220
4172024-09-07T18:50:47.400ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:224
4182024-09-07T18:50:47.400ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:226
4192024-09-07T18:50:47.400ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4202024-09-07T18:50:47.400ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4212024-09-07T18:50:47.400ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
4222024-09-07T18:50:47.441ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4232024-09-07T18:50:47.449ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_50801f60-af2f-4f08-8e46-106a71ec966e/crypt/zone file = sled-storage/src/dataset.rs:281
4242024-09-07T18:50:47.625ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_50801f60-af2f-4f08-8e46-106a71ec966e file = sled-storage/src/dataset.rs:367
4252024-09-07T18:50:47.633ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4262024-09-07T18:50:47.633ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4272024-09-07T18:50:47.633ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4282024-09-07T18:50:47.633ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 951f74f7-28ed-48ce-8b8e-a82d754c6f9c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f677c761-8af0-4ab3-b20f-2436c20df40c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f5b9b071-b292-42ff-8f22-90770e6df42a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 0b0f060f-15eb-447f-a739-7bce57c42c4b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 9d00b4ab-4f70-4f02-8fff-824dda813e40 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 78ea79f6-adbf-4571-a535-37760fc84bd0 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: cb8c5357-055f-48a8-ad91-4c8525191c8d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 50801f60-af2f-4f08-8e46-106a71ec966e (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4292024-09-07T18:50:47.633ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4302024-09-07T18:50:47.633ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4312024-09-07T18:50:47.633ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4322024-09-07T18:50:47.634ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:860 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4332024-09-07T18:50:47.634ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4342024-09-07T18:50:47.634ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:886 generation = Generation(1) sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4352024-09-07T18:50:47.634ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:891 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4362024-09-07T18:50:47.634ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:896 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4372024-09-07T18:50:47.634ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4382024-09-07T18:50:47.634ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:902 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4392024-09-07T18:50:47.634ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:803 gen = Generation(1)
4402024-09-07T18:50:47.634ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:907 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4412024-09-07T18:50:47.634ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3455560 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53635 req_id = 5204ccf4-7921-4814-b4a2-1a7311ae42ea response_code = 200 uri = /omicron-physical-disks
4422024-09-07T18:50:47.665ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 34fef005-d46a-42e7-b712-276d2ec90304, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: de009927-0679-4b92-8b0c-f90a11fb5dc2, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4432024-09-07T18:50:47.666ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34714
4442024-09-07T18:50:47.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 34fef005-d46a-42e7-b712-276d2ec90304, dataset_name: DatasetName { pool_name: ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4452024-09-07T18:50:47.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 34fef005-d46a-42e7-b712-276d2ec90304, dataset_name: DatasetName { pool_name: ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4462024-09-07T18:50:47.715ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/da86b797-b25a-4724-8ccb-680ac8134295): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4472024-09-07T18:50:47.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: de009927-0679-4b92-8b0c-f90a11fb5dc2, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4482024-09-07T18:50:47.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: de009927-0679-4b92-8b0c-f90a11fb5dc2, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4492024-09-07T18:50:47.771ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83, dataset_name: DatasetName { pool_name: ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4502024-09-07T18:50:47.771ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83, dataset_name: DatasetName { pool_name: ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4512024-09-07T18:50:47.820ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4522024-09-07T18:50:47.820ZINFOSledAgent (ServiceManager): No ledger in /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/omicron-zones.json file = common/src/ledger.rs:188
4532024-09-07T18:50:47.820ZINFOSledAgent (ServiceManager): No ledger in /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/omicron-zones.json file = common/src/ledger.rs:188
4542024-09-07T18:50:47.820ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4552024-09-07T18:50:47.864ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4562024-09-07T18:50:47.864ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4572024-09-07T18:50:47.864ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4582024-09-07T18:50:47.916ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304 file = illumos-utils/src/zone.rs:316
4592024-09-07T18:50:47.920ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83 file = illumos-utils/src/zone.rs:316
4602024-09-07T18:50:47.923ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2 file = illumos-utils/src/zone.rs:316
4612024-09-07T18:50:47.936ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304 file = illumos-utils/src/zone.rs:354
4622024-09-07T18:50:47.940ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83 file = illumos-utils/src/zone.rs:354
4632024-09-07T18:50:47.944ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2 file = illumos-utils/src/zone.rs:354
4642024-09-07T18:50:48.717ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/da86b797-b25a-4724-8ccb-680ac8134295): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
4652024-09-07T18:50:49.426ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4662024-09-07T18:50:49.426ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4672024-09-07T18:50:49.904ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4682024-09-07T18:50:49.904ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4692024-09-07T18:50:50.388ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4702024-09-07T18:50:50.388ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4712024-09-07T18:50:50.388ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4722024-09-07T18:50:50.388ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
4732024-09-07T18:50:50.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 42.994328ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4742024-09-07T18:50:50.402ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4752024-09-07T18:50:50.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 76.262247ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4762024-09-07T18:50:50.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 33.524965ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
4772024-09-07T18:50:50.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 29.803651ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4782024-09-07T18:50:50.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 245.000682ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4792024-09-07T18:50:50.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 125.882768ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
4802024-09-07T18:50:50.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 118.798998ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4812024-09-07T18:50:50.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 294.90481ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4822024-09-07T18:50:50.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 150.85861ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
4832024-09-07T18:50:50.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 350.175412ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4842024-09-07T18:50:50.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 482.842063ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
4852024-09-07T18:50:51.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 499.253504ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4862024-09-07T18:50:51.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 1.199606165s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4872024-09-07T18:50:51.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 855.017558ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
4882024-09-07T18:50:51.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 487.972059ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4892024-09-07T18:50:52.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 1.084183848s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4902024-09-07T18:50:52.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 1.379917989s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
4912024-09-07T18:50:52.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 1.083369259s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4922024-09-07T18:50:53.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 1.25866393s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
4932024-09-07T18:50:53.324ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4187
4942024-09-07T18:50:53.324ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4197 gateway = fd00:1122:3344:101::1
4952024-09-07T18:50:53.330ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4276
4962024-09-07T18:50:53.449ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4324
4972024-09-07T18:50:53.449ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4351
4982024-09-07T18:50:53.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 720.920645ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
4992024-09-07T18:50:53.488ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4364
5002024-09-07T18:50:53.488ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4228
5012024-09-07T18:50:53.542ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4270
5022024-09-07T18:50:53.542ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4331 rack_subnet = fd00:1122:3344::
5032024-09-07T18:50:53.561ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4342
5042024-09-07T18:50:53.562ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4384
5052024-09-07T18:50:53.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 1.16203368s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
5062024-09-07T18:50:53.649ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4416
5072024-09-07T18:50:53.649ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4422
5082024-09-07T18:50:53.757ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4463
5092024-09-07T18:50:53.757ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5102024-09-07T18:50:53.791ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/da86b797-b25a-4724-8ccb-680ac8134295): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:571 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
5112024-09-07T18:50:53.791ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 41.21435ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5122024-09-07T18:50:53.833ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 144.584285ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5132024-09-07T18:50:53.979ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 143.57596ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5142024-09-07T18:50:54.125ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5152024-09-07T18:50:54.158ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5162024-09-07T18:50:54.159ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5172024-09-07T18:50:54.159ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5182024-09-07T18:50:54.159ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5192024-09-07T18:50:54.196ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:42217
5202024-09-07T18:50:54.196ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:42217 req_id = f2cecdb6-603c-4361-8680-de122182769c response_code = 200 uri = /sled-identifiers
5212024-09-07T18:50:54.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 913.088866ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
5222024-09-07T18:50:54.203ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
5232024-09-07T18:50:54.255ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5242024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5252024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5262024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5272024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5282024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5292024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5302024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5312024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5322024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5332024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5342024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5352024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5362024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5372024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5382024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5392024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5402024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5412024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5422024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5432024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5442024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5452024-09-07T18:50:54.255ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5462024-09-07T18:50:54.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83") failed: Property not found. retry in 1.173496415s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
5472024-09-07T18:50:54.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2") failed: Property not found. retry in 1.127046062s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
5482024-09-07T18:50:55.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304") failed: Property not found. retry in 630.661055ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
5492024-09-07T18:50:55.964ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_de009927-0679-4b92-8b0c-f90a11fb5dc2
5502024-09-07T18:50:55.964ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_34fef005-d46a-42e7-b712-276d2ec90304
5512024-09-07T18:50:55.964ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83
5522024-09-07T18:50:55.964ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5532024-09-07T18:50:55.964ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5542024-09-07T18:50:55.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8298823 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34714 req_id = 7a25517f-6838-43fb-a6bd-b7c0cf6fb956 response_code = 204 uri = /omicron-zones
5552024-09-07T18:50:55.966ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5562024-09-07T18:50:56.014ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5572024-09-07T18:50:56.070ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5582024-09-07T18:50:56.115ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5592024-09-07T18:50:56.116ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5602024-09-07T18:50:56.150ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5612024-09-07T18:50:56.151ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5622024-09-07T18:50:56.151ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5632024-09-07T18:50:56.180ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 34fef005-d46a-42e7-b712-276d2ec90304, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: de009927-0679-4b92-8b0c-f90a11fb5dc2, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 434193fa-4a13-4c27-8229-3a8b4e208861, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6ba9518e-b713-4094-ae2c-ac769b602c51, kind: Service { id: 434193fa-4a13-4c27-8229-3a8b4e208861 }, name: Name("ntp-434193fa-4a13-4c27-8229-3a8b4e208861"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 208, 72])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5642024-09-07T18:50:56.181ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38331
5652024-09-07T18:50:56.181ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 34fef005-d46a-42e7-b712-276d2ec90304, dataset_name: DatasetName { pool_name: ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5662024-09-07T18:50:56.181ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 34fef005-d46a-42e7-b712-276d2ec90304, dataset_name: DatasetName { pool_name: ZpoolName { id: 308c8624-6854-4c09-8fdb-403d3e7020c4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5672024-09-07T18:50:56.200ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: de009927-0679-4b92-8b0c-f90a11fb5dc2, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5682024-09-07T18:50:56.201ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: de009927-0679-4b92-8b0c-f90a11fb5dc2, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c49d716-0b82-4096-a83b-a43dd8cb57a5 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5692024-09-07T18:50:56.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83, dataset_name: DatasetName { pool_name: ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5702024-09-07T18:50:56.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: af9e1e4c-5a5a-45bb-9c51-f2806bc3ae83, dataset_name: DatasetName { pool_name: ZpoolName { id: faa0cc11-3e46-4294-868c-12831b857884 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5712024-09-07T18:50:56.231ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5722024-09-07T18:50:56.231ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/omicron-zones.json file = common/src/ledger.rs:177
5732024-09-07T18:50:56.232ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/omicron-zones.json file = common/src/ledger.rs:177
5742024-09-07T18:50:56.232ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5752024-09-07T18:50:56.242ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5762024-09-07T18:50:56.275ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5772024-09-07T18:50:56.277ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5782024-09-07T18:50:56.277ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5792024-09-07T18:50:56.308ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 208, 72]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5802024-09-07T18:50:56.308ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = boundary_ntp
5812024-09-07T18:50:56.309ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5822024-09-07T18:50:56.309ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5832024-09-07T18:50:56.309ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5842024-09-07T18:50:56.310ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5852024-09-07T18:50:56.334ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861 file = illumos-utils/src/zone.rs:316
5862024-09-07T18:50:56.351ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861 file = illumos-utils/src/zone.rs:354
5872024-09-07T18:50:56.975ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5882024-09-07T18:50:56.975ZINFOSledAgent (ServiceManager): Booting oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5892024-09-07T18:50:57.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 25.110846ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5902024-09-07T18:50:57.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 110.626158ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5912024-09-07T18:50:57.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 195.262244ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5922024-09-07T18:50:57.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 368.131268ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5932024-09-07T18:50:57.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 691.002695ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5942024-09-07T18:50:58.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 695.334302ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5952024-09-07T18:50:59.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 787.478141ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5962024-09-07T18:51:00.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 1.125612122s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5972024-09-07T18:51:00.795ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/da86b797-b25a-4724-8ccb-680ac8134295): error trying to connect: dns error: no record found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } file = sled-agent/src/nexus.rs:571 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
5982024-09-07T18:51:01.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861") failed: Property not found. retry in 1.231081086s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
5992024-09-07T18:51:02.455ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861
6002024-09-07T18:51:02.455ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/951f74f7-28ed-48ce-8b8e-a82d754c6f9c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6012024-09-07T18:51:02.455ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f677c761-8af0-4ab3-b20f-2436c20df40c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6022024-09-07T18:51:02.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6274940 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38331 req_id = 41ab6306-6d07-4322-918d-68001a649050 response_code = 204 uri = /omicron-zones
6032024-09-07T18:51:02.458ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6042024-09-07T18:51:02.488ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6052024-09-07T18:51:02.488ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41731
6062024-09-07T18:51:02.498ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6072024-09-07T18:51:02.498ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 9014 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41731 req_id = 993415f6-b60e-43ce-af7e-8ca47b5bee17 response_code = 500 uri = /timesync
6082024-09-07T18:51:02.500ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6092024-09-07T18:51:02.663ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6102024-09-07T18:51:02.663ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56130
6112024-09-07T18:51:06.671ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6122024-09-07T18:51:06.671ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4007945 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56130 req_id = 3dfbd346-a39c-4b31-9eba-b26fe0c0d678 response_code = 500 uri = /timesync
6132024-09-07T18:51:06.674ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6142024-09-07T18:51:06.851ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6152024-09-07T18:51:06.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48677
6162024-09-07T18:51:06.860ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6172024-09-07T18:51:06.860ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8511 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48677 req_id = 6a164255-b277-48b8-bef4-e231fe44d6d2 response_code = 500 uri = /timesync
6182024-09-07T18:51:06.861ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6192024-09-07T18:51:07.093ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6202024-09-07T18:51:07.093ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48940
6212024-09-07T18:51:07.102ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_434193fa-4a13-4c27-8229-3a8b4e208861': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6222024-09-07T18:51:07.102ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8828 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48940 req_id = a9474b3f-ac68-481c-919e-a9bd7e6e0958 response_code = 500 uri = /timesync
6232024-09-07T18:51:07.104ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6242024-09-07T18:51:07.317ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6252024-09-07T18:51:07.317ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37584
6262024-09-07T18:51:11.326ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4008844 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37584 req_id = 9fb6e501-77b2-49a2-b356-bf3b235784a2 response_code = 200 uri = /timesync
6272024-09-07T18:51:11.329ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6282024-09-07T18:51:11.329ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6292024-09-07T18:51:11.645ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6302024-09-07T18:51:11.645ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36753
6312024-09-07T18:51:11.654ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36753 req_id = b54e7979-419d-4ec3-bde9-bc1979b52dfe response_code = 200 uri = /timesync
6322024-09-07T18:51:11.657ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6332024-09-07T18:51:11.657ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6342024-09-07T18:51:11.965ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6352024-09-07T18:51:11.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40954
6362024-09-07T18:51:11.973ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8044 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40954 req_id = d67861d4-239d-4662-9e65-74e1ebe1bf7c response_code = 200 uri = /timesync
6372024-09-07T18:51:11.975ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6382024-09-07T18:51:11.975ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6392024-09-07T18:51:12.303ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6402024-09-07T18:51:12.303ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49992
6412024-09-07T18:51:12.311ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7877 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49992 req_id = b9997b2e-941b-4402-acf8-9e1650165ae4 response_code = 200 uri = /timesync
6422024-09-07T18:51:12.313ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6432024-09-07T18:51:12.313ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6442024-09-07T18:51:12.856ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6452024-09-07T18:51:12.856ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40707
6462024-09-07T18:51:12.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7604 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40707 req_id = dfdd2bc3-ac5a-4ce9-8d54-5323c0b195b0 response_code = 200 uri = /timesync
6472024-09-07T18:51:12.867ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6482024-09-07T18:51:12.867ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6492024-09-07T18:51:13.274ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6502024-09-07T18:51:13.274ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40398
6512024-09-07T18:51:13.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40398 req_id = 8b55de6d-5b12-4ecb-a3cd-fae0848579a6 response_code = 200 uri = /timesync
6522024-09-07T18:51:13.283ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6532024-09-07T18:51:13.283ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6542024-09-07T18:51:14.079ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6552024-09-07T18:51:14.080ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50629
6562024-09-07T18:51:14.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8273 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50629 req_id = 1a69fc9d-52c3-45da-bbbd-3b10cb0095af response_code = 200 uri = /timesync
6572024-09-07T18:51:14.090ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6582024-09-07T18:51:14.090ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6592024-09-07T18:51:14.448ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6602024-09-07T18:51:14.448ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59832
6612024-09-07T18:51:14.457ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8655 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59832 req_id = 6a677db1-32de-4a89-83c1-e29cffadfd88 response_code = 200 uri = /timesync
6622024-09-07T18:51:14.459ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6632024-09-07T18:51:14.459ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6642024-09-07T18:51:14.855ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 368.11413ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
6652024-09-07T18:51:15.286ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6662024-09-07T18:51:15.286ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42585
6672024-09-07T18:51:15.295ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8578 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42585 req_id = eff7152e-7828-4a3a-9811-fc8b86a809e4 response_code = 200 uri = /timesync
6682024-09-07T18:51:15.297ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6692024-09-07T18:51:15.297ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6702024-09-07T18:51:16.034ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6712024-09-07T18:51:16.035ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36281
6722024-09-07T18:51:16.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8881 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36281 req_id = fc01ad9f-da2e-40bd-993e-a2985361dd7b response_code = 200 uri = /timesync
6732024-09-07T18:51:16.045ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6742024-09-07T18:51:16.045ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6752024-09-07T18:51:16.836ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6762024-09-07T18:51:16.836ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55862
6772024-09-07T18:51:16.844ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8319 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55862 req_id = 438769e5-7dcc-46bf-8a79-8c6cbffea393 response_code = 200 uri = /timesync
6782024-09-07T18:51:16.848ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6792024-09-07T18:51:16.848ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6802024-09-07T18:51:17.771ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6812024-09-07T18:51:17.771ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39173
6822024-09-07T18:51:17.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8556 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39173 req_id = 3cab679e-03c5-436d-ad6c-04d13545a28d response_code = 200 uri = /timesync
6832024-09-07T18:51:17.783ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6842024-09-07T18:51:17.783ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6852024-09-07T18:51:19.495ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6862024-09-07T18:51:19.495ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47071
6872024-09-07T18:51:19.504ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8228 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47071 req_id = a0667ce2-56c3-4b15-ade9-e99ccd615c20 response_code = 200 uri = /timesync
6882024-09-07T18:51:19.506ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6892024-09-07T18:51:19.506ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6902024-09-07T18:51:21.570ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6912024-09-07T18:51:21.570ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55442
6922024-09-07T18:51:21.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9473 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55442 req_id = 156e5aaa-3dc7-4abc-8ef4-0fa74e0136c3 response_code = 200 uri = /timesync
6932024-09-07T18:51:21.582ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6942024-09-07T18:51:21.582ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6952024-09-07T18:51:23.533ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:40063
6962024-09-07T18:51:23.533ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 54 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:40063 req_id = f9b883d1-86fc-41f0-a23d-4f2ac8b3204c response_code = 200 uri = /baseboard
6972024-09-07T18:51:24.032ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6982024-09-07T18:51:24.032ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35211
6992024-09-07T18:51:24.041ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8859 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35211 req_id = ed142a58-9da7-4e54-a05f-6f968955d083 response_code = 200 uri = /timesync
7002024-09-07T18:51:24.045ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7012024-09-07T18:51:24.045ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7022024-09-07T18:51:26.568ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7032024-09-07T18:51:26.568ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63802
7042024-09-07T18:51:26.577ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8754 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63802 req_id = 3bb65bca-fb67-4923-aec8-cf32e669dd40 response_code = 200 uri = /timesync
7052024-09-07T18:51:26.579ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7062024-09-07T18:51:26.579ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7072024-09-07T18:51:28.737ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7082024-09-07T18:51:28.737ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43174
7092024-09-07T18:51:28.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8957 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43174 req_id = d0fb87e7-0b05-43c9-b644-3c91758a0126 response_code = 200 uri = /timesync
7102024-09-07T18:51:28.750ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7112024-09-07T18:51:28.750ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7122024-09-07T18:51:30.259ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 414.47922ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7132024-09-07T18:51:33.186ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7142024-09-07T18:51:33.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51873
7152024-09-07T18:51:33.195ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9391 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51873 req_id = 1d61e761-5f3c-496d-9a94-f0c9c58c9df4 response_code = 200 uri = /timesync
7162024-09-07T18:51:33.198ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7172024-09-07T18:51:33.198ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7182024-09-07T18:51:38.714ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7192024-09-07T18:51:38.714ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41651
7202024-09-07T18:51:38.723ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9095 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41651 req_id = d8bc4e09-9f17-4997-a220-d20a53d7e5df response_code = 200 uri = /timesync
7212024-09-07T18:51:38.726ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7222024-09-07T18:51:38.726ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7232024-09-07T18:51:44.632ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7242024-09-07T18:51:44.632ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64226
7252024-09-07T18:51:44.640ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8473 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64226 req_id = 3331ac4a-bedc-4957-ac7d-40b75c196816 response_code = 200 uri = /timesync
7262024-09-07T18:51:44.643ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7272024-09-07T18:51:44.643ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7282024-09-07T18:51:45.708ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.143934111s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7292024-09-07T18:51:51.792ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7302024-09-07T18:51:51.792ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46570
7312024-09-07T18:51:51.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8867 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46570 req_id = 164e9474-92c3-4f29-806b-4b23b82849df response_code = 200 uri = /timesync
7322024-09-07T18:51:51.803ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7332024-09-07T18:51:51.804ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7342024-09-07T18:51:53.567ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:34453
7352024-09-07T18:51:53.567ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 51 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:34453 req_id = 8a57604b-2e60-4e6d-9bf2-cbe616931451 response_code = 200 uri = /baseboard
7362024-09-07T18:52:01.887ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.468292428s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7372024-09-07T18:52:02.205ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7382024-09-07T18:52:02.205ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57239
7392024-09-07T18:52:02.214ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8963 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57239 req_id = ecaf3efc-1f96-447c-b6c3-5bad37531f0f response_code = 200 uri = /timesync
7402024-09-07T18:52:02.217ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7412024-09-07T18:52:02.217ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7422024-09-07T18:52:11.448ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7432024-09-07T18:52:11.448ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37509
7442024-09-07T18:52:11.457ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9204 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37509 req_id = 189922e6-b4ba-487f-b746-4b3cd7f05148 response_code = 200 uri = /timesync
7452024-09-07T18:52:11.460ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7462024-09-07T18:52:11.460ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7472024-09-07T18:52:18.392ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.323840656s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7482024-09-07T18:52:23.599ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:40802
7492024-09-07T18:52:23.599ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 51 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:40802 req_id = fbc7ac5d-04e2-43e6-b8f4-4fb96e6202be response_code = 200 uri = /baseboard
7502024-09-07T18:52:25.332ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7512024-09-07T18:52:25.333ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44603
7522024-09-07T18:52:25.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9559 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44603 req_id = 67d42b66-cfe3-476e-a0a6-f0ca22df29b2 response_code = 200 uri = /timesync
7532024-09-07T18:52:25.344ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7542024-09-07T18:52:25.344ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7552024-09-07T18:52:32.963ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7562024-09-07T18:52:32.963ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42988
7572024-09-07T18:52:32.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8797 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42988 req_id = 8f492c2c-d612-4efb-bc4a-3ed43acfb3de response_code = 200 uri = /timesync
7582024-09-07T18:52:32.977ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7592024-09-07T18:52:32.977ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7602024-09-07T18:52:35.751ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 7.025775571s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7612024-09-07T18:52:53.634ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:40072
7622024-09-07T18:52:53.634ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:40072 req_id = 842483e2-4d47-4c6e-9ce3-9a3938764d74 response_code = 200 uri = /baseboard
7632024-09-07T18:52:54.824ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7642024-09-07T18:52:54.824ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51895
7652024-09-07T18:52:54.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 10005 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51895 req_id = 6c2aeddf-2cd7-46e7-9d6f-1bf7625d8c5c response_code = 200 uri = /timesync
7662024-09-07T18:52:54.838ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7672024-09-07T18:52:54.838ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7682024-09-07T18:52:57.812ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.936310574s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7692024-09-07T18:53:12.518ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7702024-09-07T18:53:12.519ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61264
7712024-09-07T18:53:12.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 10149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61264 req_id = c54bdacb-1f9d-45f8-9800-492fadb23717 response_code = 200 uri = /timesync
7722024-09-07T18:53:12.531ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7732024-09-07T18:53:12.531ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7742024-09-07T18:53:22.783ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 26.29689891s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7752024-09-07T18:53:23.672ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:50692
7762024-09-07T18:53:23.673ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 959 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:50692 req_id = af3c3a6c-d1dd-4e25-86f0-53cecede0ade response_code = 200 uri = /baseboard
7772024-09-07T18:53:39.368ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7782024-09-07T18:53:39.368ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65362
7792024-09-07T18:53:39.377ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65362 req_id = 83e27053-10eb-425e-9efd-9043017c943f response_code = 200 uri = /timesync
7802024-09-07T18:53:39.380ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7812024-09-07T18:53:39.380ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7822024-09-07T18:53:53.705ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:43459
7832024-09-07T18:53:53.706ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 836 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:43459 req_id = 4f76eb7d-9f70-4e8e-b3aa-97994ee75203 response_code = 200 uri = /baseboard
7842024-09-07T18:54:04.115ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 60.174172579s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
7852024-09-07T18:54:05.951ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7862024-09-07T18:54:05.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50030
7872024-09-07T18:54:05.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8441 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50030 req_id = 1d162b1b-abc4-426a-bedf-e1d90b4a9322 response_code = 200 uri = /timesync
7882024-09-07T18:54:05.962ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7892024-09-07T18:54:05.962ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7902024-09-07T18:54:23.743ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:35119
7912024-09-07T18:54:23.743ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 798 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:35119 req_id = 5111a5ca-689a-4f76-a8f5-7325b34f6f4a response_code = 200 uri = /baseboard
7922024-09-07T18:54:48.050ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7932024-09-07T18:54:48.050ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59643
7942024-09-07T18:54:48.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 10316 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59643 req_id = 6b81a298-98ce-4f9a-b5a6-c63391ce011b response_code = 200 uri = /timesync
7952024-09-07T18:54:48.063ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
7962024-09-07T18:54:48.063ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7972024-09-07T18:54:53.775ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:33174
7982024-09-07T18:54:53.776ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 62 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:33174 req_id = f308cfd3-2fd3-4a4b-b43b-179bfa2b6f53 response_code = 200 uri = /baseboard
7992024-09-07T18:55:19.326ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 181.438348896s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = da86b797-b25a-4724-8ccb-680ac8134295
8002024-09-07T18:55:23.808ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:50966
8012024-09-07T18:55:23.808ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:50966 req_id = a4349bfc-99f9-4fc5-9408-e07a656ba494 response_code = 200 uri = /baseboard
8022024-09-07T18:55:35.918ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8032024-09-07T18:55:35.918ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34862
8042024-09-07T18:55:35.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9610 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34862 req_id = c13dd88e-57fd-4bfc-a40a-c09b78b93b69 response_code = 200 uri = /timesync
8052024-09-07T18:55:35.930ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
8062024-09-07T18:55:35.930ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8072024-09-07T18:55:47.634ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
8082024-09-07T18:55:53.840ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:43444
8092024-09-07T18:55:53.840ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:43444 req_id = adc68a25-520c-48a5-a070-be60a694b143 response_code = 200 uri = /baseboard