Max level shown:
1 [ Sep 6 06:33:20 Enabled. ]
2 [ Sep 6 06:33:20 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Sep 6 06:33:20 Method "start" exited with status 0. ]
4 [ Sep 6 06:33:20 Rereading configuration. ]
5 [ Sep 6 06:33:20 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-09-06T06:33:20.878ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-09-06T06:33:20.878ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-09-06T06:33:20.930ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-09-06T06:33:20.930ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-09-06T06:33:21.045ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-09-06T06:33:21.054ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-09-06T06:33:21.062ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-09-06T06:33:21.664ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-09-06T06:33:21.802ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-09-06T06:33:21.802ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-09-06T06:33:21.802ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-09-06T06:33:21.802ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-09-06T06:33:21.802ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-09-06T06:33:21.802ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-09-06T06:33:21.805ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-09-06T06:33:21.805ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-09-06T06:33:21.805ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-09-06T06:33:21.805ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-09-06T06:33:21.805ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-09-06T06:33:21.805ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-09-06T06:33:21.805ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-09-06T06:33:21.805ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-09-06T06:33:21.805ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-09-06T06:33:21.816ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-09-06T06:33:21.816ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool)
322024-09-06T06:33:21.917ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal }
332024-09-06T06:33:22.320ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-09-06T06:33:22.321ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-09-06T06:33:22.321ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-09-06T06:33:22.321ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-09-06T06:33:22.321ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-09-06T06:33:22.321ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-09-06T06:33:22.321ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-09-06T06:33:22.329ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-09-06T06:33:22.329ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool)
422024-09-06T06:33:22.415ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal }
432024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
522024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
532024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
542024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
582024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
602024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
762024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
772024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
782024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
822024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
882024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
892024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
902024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-09-06T06:33:22.819ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
932024-09-06T06:33:22.819ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
942024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
952024-09-06T06:33:22.819ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
962024-09-06T06:33:22.819ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-09-06T06:33:22.819ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-09-06T06:33:22.819ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-09-06T06:33:22.819ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-09-06T06:33:22.820ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-09-06T06:33:22.820ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-09-06T06:33:22.820ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1032024-09-06T06:33:22.820ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-09-06T06:33:22.820ZINFOSledAgent (bootstore): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-09-06T06:33:22.820ZINFOSledAgent (bootstore): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-09-06T06:33:22.820ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-09-06T06:33:22.820ZINFOSledAgent (bootstore): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-09-06T06:33:22.820ZINFOSledAgent (bootstore): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-09-06T06:33:22.820ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-09-06T06:33:22.820ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-09-06T06:33:22.820ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-09-06T06:33:22.830ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:730
1132024-09-06T06:33:22.830ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 7226, tv_nsec: 692263539 } time_to_next_cleanup = 599.999919349s
1142024-09-06T06:33:22.830ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3842
1152024-09-06T06:33:22.830ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-09-06T06:33:22.830ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-09-06T06:33:22.830ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-09-06T06:33:22.833ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-09-06T06:33:22.833ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-09-06T06:33:22.833ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-09-06T06:33:22.833ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1222024-09-06T06:33:22.833ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-09-06T06:33:22.834ZINFOSledAgent (RSS): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-09-06T06:33:22.834ZINFOSledAgent (RSS): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-09-06T06:33:22.834ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-09-06T06:33:22.834ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-09-06T06:33:22.834ZINFOSledAgent (RSS): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-09-06T06:33:22.834ZINFOSledAgent (RSS): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-09-06T06:33:22.834ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-09-06T06:33:22.834ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1312024-09-06T06:33:22.834ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1322024-09-06T06:33:22.834ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-09-06T06:33:22.834ZINFOSledAgent (RSS): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-09-06T06:33:22.834ZINFOSledAgent (RSS): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-09-06T06:33:22.834ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 99cd7c46-10a2-48c7-8118-fbe80ea47643, sleds: { [fdb0:18c0:4d0d:9fb2::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 58853ca1-8d5a-4912-87e6-57de713a602c, rack_id: 99cd7c46-10a2-48c7-8118-fbe80ea47643, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1362024-09-06T06:33:22.834ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-09-06T06:33:22.834ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-09-06T06:33:22.834ZINFOSledAgent (bootstore): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-09-06T06:33:22.834ZINFOSledAgent (bootstore): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-09-06T06:33:22.835ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-09-06T06:33:22.835ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:221 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 58853ca1-8d5a-4912-87e6-57de713a602c, rack_id: 99cd7c46-10a2-48c7-8118-fbe80ea47643, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-09-06T06:33:22.835ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:61291
1432024-09-06T06:33:22.835ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 58853ca1-8d5a-4912-87e6-57de713a602c, rack_id: 99cd7c46-10a2-48c7-8118-fbe80ea47643, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-09-06T06:33:22.835ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1462024-09-06T06:33:22.835ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1472024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1482024-09-06T06:33:22.835ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1492024-09-06T06:33:22.835ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1502024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1512024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1522024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1532024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1542024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1552024-09-06T06:33:22.835ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1562024-09-06T06:33:22.835ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1572024-09-06T06:33:22.837ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4139
1582024-09-06T06:33:22.837ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4521
1592024-09-06T06:33:22.867ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1602024-09-06T06:33:22.888ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-09-06T06:33:22.905ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-09-06T06:33:22.915ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:390 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1632024-09-06T06:33:22.915ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1642024-09-06T06:33:22.915ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:402 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1652024-09-06T06:33:22.915ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-09-06T06:33:22.918ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 56.202179ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c870f6dc-0316-4041-b9fd-eb2e597a389a", "content-length": "124", "date": "Fri, 06 Sep 2024 06:33:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c870f6dc-0316-4041-b9fd-eb2e597a389a" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-09-06T06:33:22.928ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_acfe4e3e-8cb9-4336-9553-a7677bbf6061/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-09-06T06:33:22.951ZINFOSledAgent: successfully created encrypted zvol: name="oxi_acfe4e3e-8cb9-4336-9553-a7677bbf6061/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-09-06T06:33:22.951ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_acfe4e3e-8cb9-4336-9553-a7677bbf6061/swap" file = sled-agent/src/swap_device.rs:96
1702024-09-06T06:33:22.956ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:417 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1712024-09-06T06:33:22.956ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:130
1722024-09-06T06:33:22.956ZINFOSledAgent (BackingFs): Ensuring dataset oxi_acfe4e3e-8cb9-4336-9553-a7677bbf6061/backing/fmd file = sled-agent/src/backing_fs.rs:140
1732024-09-06T06:33:22.975ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-09-06T06:33:22.976ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 167.646813ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c1aa23e5-c97b-4467-a2dd-e17c6a747770", "content-length": "124", "date": "Fri, 06 Sep 2024 06:33:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c1aa23e5-c97b-4467-a2dd-e17c6a747770" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-09-06T06:33:23.040ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:175
1762024-09-06T06:33:23.145ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-09-06T06:33:23.145ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 130.773703ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d94707ce-4f92-4682-afa5-4b1d4f3ff2bd", "content-length": "124", "date": "Fri, 06 Sep 2024 06:33:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d94707ce-4f92-4682-afa5-4b1d4f3ff2bd" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-09-06T06:33:23.259ZINFOSledAgent (BackingFs): Mounting oxi_acfe4e3e-8cb9-4336-9553-a7677bbf6061/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:183
1792024-09-06T06:33:23.272ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:191
1802024-09-06T06:33:23.272ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:191
1812024-09-06T06:33:23.272ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:191
1822024-09-06T06:33:23.272ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:197
1832024-09-06T06:33:23.278ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-09-06T06:33:23.278ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 128.957275ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "33c4e529-07c9-4d80-aa42-bcef2a0d87b9", "content-length": "124", "date": "Fri, 06 Sep 2024 06:33:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "33c4e529-07c9-4d80-aa42-bcef2a0d87b9" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-09-06T06:33:23.407ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-09-06T06:33:23.410ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 264.752656ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "213c506d-200a-48ad-b5d1-eec90d4e64f7", "content-length": "124", "date": "Fri, 06 Sep 2024 06:33:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "213c506d-200a-48ad-b5d1-eec90d4e64f7" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-09-06T06:33:23.674ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-09-06T06:33:23.674ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 296.690154ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7fac9b67-e323-4bc5-8a23-569927314bef", "content-length": "124", "date": "Fri, 06 Sep 2024 06:33:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7fac9b67-e323-4bc5-8a23-569927314bef" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-09-06T06:33:23.972ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1902024-09-06T06:33:23.999ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1912024-09-06T06:33:24.004ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:38689 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1922024-09-06T06:33:24.004ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1932024-09-06T06:33:24.005ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:38689 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 58853ca1-8d5a-4912-87e6-57de713a602c sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1942024-09-06T06:33:24.006ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 142.06265ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1952024-09-06T06:33:24.006ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1962024-09-06T06:33:24.006ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
1972024-09-06T06:33:24.006ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:914 underlay_address = fd00:1122:3344:101::1
1982024-09-06T06:33:24.006ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1992024-09-06T06:33:24.149ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 749.515394ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2002024-09-06T06:33:24.900ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.257201338s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2012024-09-06T06:33:26.160ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.152750512s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2022024-09-06T06:33:26.490ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2615
2032024-09-06T06:33:26.490ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2833
2042024-09-06T06:33:26.490ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2485
2052024-09-06T06:33:26.490ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2532
2062024-09-06T06:33:26.490ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2914
2072024-09-06T06:33:26.490ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2956
2082024-09-06T06:33:26.492ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2758
2092024-09-06T06:33:26.493ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2526
2102024-09-06T06:33:26.494ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='f9f334d9-15b1-42c2-84a8-de0b4d4370bf'/> <propval type="astring" name="rack_id" value='99cd7c46-10a2-48c7-8118-fbe80ea47643'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe0c:eb20'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='58853ca1-8d5a-4912-87e6-57de713a602c'/> <propval type="astring" name="rack_id" value='99cd7c46-10a2-48c7-8118-fbe80ea47643'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='58853ca1-8d5a-4912-87e6-57de713a602c'/> <propval type="astring" name="rack_uuid" value='99cd7c46-10a2-48c7-8118-fbe80ea47643'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='58853ca1-8d5a-4912-87e6-57de713a602c'/> <propval type="astring" name="rack_uuid" value='99cd7c46-10a2-48c7-8118-fbe80ea47643'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2112024-09-06T06:33:26.494ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2122024-09-06T06:33:26.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 37.54514ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2132024-09-06T06:33:27.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 70.402257ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2142024-09-06T06:33:27.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 105.082387ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2152024-09-06T06:33:27.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 290.903783ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2162024-09-06T06:33:27.379ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.865570996s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2172024-09-06T06:33:27.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.023645704s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2182024-09-06T06:33:29.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 888.258423ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-09-06T06:33:29.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.013664392s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-09-06T06:33:30.247ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 7.604626582s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2212024-09-06T06:33:31.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.328863996s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-09-06T06:33:32.931ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2232024-09-06T06:33:32.931ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2242024-09-06T06:33:32.931ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2252024-09-06T06:33:32.931ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2262024-09-06T06:33:32.931ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2272024-09-06T06:33:32.932ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2282024-09-06T06:33:32.932ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:61291
2292024-09-06T06:33:32.932ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2302024-09-06T06:33:32.932ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3842
2312024-09-06T06:33:32.932ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:112 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2322024-09-06T06:33:32.932ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:236 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2332024-09-06T06:33:33.163ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2342024-09-06T06:33:33.163ZINFOSledAgent (RSS): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2352024-09-06T06:33:33.163ZINFOSledAgent (RSS): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2362024-09-06T06:33:33.163ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2372024-09-06T06:33:33.163ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2382024-09-06T06:33:33.163ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2392024-09-06T06:33:33.168ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4163 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2402024-09-06T06:33:33.168ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4179
2412024-09-06T06:33:33.168ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2422024-09-06T06:33:33.431ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62217
2432024-09-06T06:33:33.431ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-09-06T06:33:33.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 297 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62217 req_id = 0391aebe-6a86-49d5-8111-13a9e9a96d30 response_code = 200 uri = /inventory
2452024-09-06T06:33:33.462ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45457
2462024-09-06T06:33:33.463ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45457 req_id = dc7bc9f0-d1cd-4ed3-97f1-e617ac97b798 response_code = 200 uri = /sled-role
2472024-09-06T06:33:33.464ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2482024-09-06T06:33:33.464ZINFOSledAgent (RSS): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2492024-09-06T06:33:33.465ZINFOSledAgent (RSS): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2502024-09-06T06:33:33.465ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:1001
2512024-09-06T06:33:33.495ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: eb7a6830-bf94-441c-aad1-3545b6f9459b, pool_id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 0e631d89-c569-4888-ba01-7ef5f6a2b898, pool_id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 39df5511-145e-4e13-8e99-c09d032c1843, pool_id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 31a35d21-f38e-4360-ac91-02ff8e644a61, pool_id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 8d20c6e7-9132-4f38-a6e6-f51328fcbb12, pool_id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 393043dc-507d-4c62-95c6-12d1d4b9946b, pool_id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 983be032-0f92-40e4-a906-2dc6c24dd5f1, pool_id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 27b92b81-dc36-4a53-8cdd-ffadfdb6f6f3, pool_id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 6aa32561-7ad8-4ffd-8c32-cdeb79234bf0, pool_id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2522024-09-06T06:33:33.495ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48098
2532024-09-06T06:33:33.496ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:858 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2542024-09-06T06:33:33.496ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: eb7a6830-bf94-441c-aad1-3545b6f9459b, pool_id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 0e631d89-c569-4888-ba01-7ef5f6a2b898, pool_id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 39df5511-145e-4e13-8e99-c09d032c1843, pool_id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 31a35d21-f38e-4360-ac91-02ff8e644a61, pool_id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 8d20c6e7-9132-4f38-a6e6-f51328fcbb12, pool_id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 393043dc-507d-4c62-95c6-12d1d4b9946b, pool_id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 983be032-0f92-40e4-a906-2dc6c24dd5f1, pool_id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 27b92b81-dc36-4a53-8cdd-ffadfdb6f6f3, pool_id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 6aa32561-7ad8-4ffd-8c32-cdeb79234bf0, pool_id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2552024-09-06T06:33:33.496ZINFOSledAgent (StorageManager): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2562024-09-06T06:33:33.496ZINFOSledAgent (StorageManager): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2572024-09-06T06:33:33.496ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2582024-09-06T06:33:33.496ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2592024-09-06T06:33:33.496ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2602024-09-06T06:33:33.496ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2612024-09-06T06:33:33.496ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2622024-09-06T06:33:33.676ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2632024-09-06T06:33:33.676ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool)
2642024-09-06T06:33:34.144ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/58853ca1-8d5a-4912-87e6-57de713a602c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2652024-09-06T06:33:34.607ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2662024-09-06T06:33:34.638ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }
2672024-09-06T06:33:34.646ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:229
2682024-09-06T06:33:34.646ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:231
2692024-09-06T06:33:34.646ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:235
2702024-09-06T06:33:34.646ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:237
2712024-09-06T06:33:34.646ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2722024-09-06T06:33:34.646ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2732024-09-06T06:33:34.646ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
2742024-09-06T06:33:34.681ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2752024-09-06T06:33:34.689ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7516b8f4-123a-4b07-8c44-30d87eec7e9c/crypt/zone file = sled-storage/src/dataset.rs:292
2762024-09-06T06:33:34.860ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7516b8f4-123a-4b07-8c44-30d87eec7e9c file = sled-storage/src/dataset.rs:378
2772024-09-06T06:33:34.867ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2782024-09-06T06:33:34.867ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2792024-09-06T06:33:34.867ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2802024-09-06T06:33:34.867ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2812024-09-06T06:33:34.867ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2822024-09-06T06:33:34.877ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2832024-09-06T06:33:34.877ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool)
2842024-09-06T06:33:34.971ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }
2852024-09-06T06:33:34.979ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:229
2862024-09-06T06:33:34.979ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:231
2872024-09-06T06:33:34.979ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:235
2882024-09-06T06:33:34.979ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:237
2892024-09-06T06:33:34.979ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2902024-09-06T06:33:34.979ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2912024-09-06T06:33:34.979ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
2922024-09-06T06:33:35.015ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2932024-09-06T06:33:35.025ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_56647842-7219-46e1-9978-bdffa4d4e8cf/crypt/zone file = sled-storage/src/dataset.rs:292
2942024-09-06T06:33:35.147ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/58853ca1-8d5a-4912-87e6-57de713a602c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
2952024-09-06T06:33:35.206ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_56647842-7219-46e1-9978-bdffa4d4e8cf file = sled-storage/src/dataset.rs:378
2962024-09-06T06:33:35.214ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
2972024-09-06T06:33:35.215ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
2982024-09-06T06:33:35.215ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
2992024-09-06T06:33:35.215ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3002024-09-06T06:33:35.215ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3012024-09-06T06:33:35.224ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3022024-09-06T06:33:35.224ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool)
3032024-09-06T06:33:35.317ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }
3042024-09-06T06:33:35.325ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:229
3052024-09-06T06:33:35.325ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:231
3062024-09-06T06:33:35.325ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:235
3072024-09-06T06:33:35.325ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:237
3082024-09-06T06:33:35.325ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3092024-09-06T06:33:35.325ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3102024-09-06T06:33:35.325ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3112024-09-06T06:33:35.360ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3122024-09-06T06:33:35.367ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7a42b808-7b1f-406b-87b7-e6aeecc73464/crypt/zone file = sled-storage/src/dataset.rs:292
3132024-09-06T06:33:35.538ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7a42b808-7b1f-406b-87b7-e6aeecc73464 file = sled-storage/src/dataset.rs:378
3142024-09-06T06:33:35.546ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3152024-09-06T06:33:35.546ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3162024-09-06T06:33:35.546ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3172024-09-06T06:33:35.546ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3182024-09-06T06:33:35.546ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3192024-09-06T06:33:35.555ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3202024-09-06T06:33:35.555ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool)
3212024-09-06T06:33:35.650ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }
3222024-09-06T06:33:35.658ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:229
3232024-09-06T06:33:35.658ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:231
3242024-09-06T06:33:35.658ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:235
3252024-09-06T06:33:35.658ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:237
3262024-09-06T06:33:35.658ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3272024-09-06T06:33:35.658ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3282024-09-06T06:33:35.658ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3292024-09-06T06:33:35.691ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3302024-09-06T06:33:35.699ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a32e2a10-4360-4704-b36b-dc63733b1b01/crypt/zone file = sled-storage/src/dataset.rs:292
3312024-09-06T06:33:35.871ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a32e2a10-4360-4704-b36b-dc63733b1b01 file = sled-storage/src/dataset.rs:378
3322024-09-06T06:33:35.879ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3332024-09-06T06:33:35.879ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3342024-09-06T06:33:35.879ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3352024-09-06T06:33:35.879ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3362024-09-06T06:33:35.879ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3372024-09-06T06:33:35.889ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3382024-09-06T06:33:35.889ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool)
3392024-09-06T06:33:35.984ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }
3402024-09-06T06:33:35.992ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:229
3412024-09-06T06:33:35.992ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:231
3422024-09-06T06:33:35.992ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:235
3432024-09-06T06:33:35.992ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:237
3442024-09-06T06:33:35.992ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3452024-09-06T06:33:35.992ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3462024-09-06T06:33:35.992ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3472024-09-06T06:33:36.031ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3482024-09-06T06:33:36.039ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f881a8f5-60a1-4050-a3bb-fd18022091f4/crypt/zone file = sled-storage/src/dataset.rs:292
3492024-09-06T06:33:36.150ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/58853ca1-8d5a-4912-87e6-57de713a602c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
3502024-09-06T06:33:36.212ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f881a8f5-60a1-4050-a3bb-fd18022091f4 file = sled-storage/src/dataset.rs:378
3512024-09-06T06:33:36.220ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3522024-09-06T06:33:36.220ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3532024-09-06T06:33:36.220ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3542024-09-06T06:33:36.220ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3552024-09-06T06:33:36.220ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3562024-09-06T06:33:36.229ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3572024-09-06T06:33:36.229ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool)
3582024-09-06T06:33:36.323ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool), kind: External }
3592024-09-06T06:33:36.331ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:229
3602024-09-06T06:33:36.331ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:231
3612024-09-06T06:33:36.331ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:235
3622024-09-06T06:33:36.331ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:237
3632024-09-06T06:33:36.331ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3642024-09-06T06:33:36.331ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3652024-09-06T06:33:36.331ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3662024-09-06T06:33:36.368ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3672024-09-06T06:33:36.375ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_681fcf67-79af-4c28-8e2f-62c72e00e09a/crypt/zone file = sled-storage/src/dataset.rs:292
3682024-09-06T06:33:36.556ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_681fcf67-79af-4c28-8e2f-62c72e00e09a file = sled-storage/src/dataset.rs:378
3692024-09-06T06:33:36.563ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3702024-09-06T06:33:36.563ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3712024-09-06T06:33:36.563ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3722024-09-06T06:33:36.563ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3732024-09-06T06:33:36.563ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3742024-09-06T06:33:36.573ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3752024-09-06T06:33:36.573ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool)
3762024-09-06T06:33:36.666ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool), kind: External }
3772024-09-06T06:33:36.675ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:229
3782024-09-06T06:33:36.675ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:231
3792024-09-06T06:33:36.675ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:235
3802024-09-06T06:33:36.675ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:237
3812024-09-06T06:33:36.675ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3822024-09-06T06:33:36.675ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3832024-09-06T06:33:36.675ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3842024-09-06T06:33:36.709ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3852024-09-06T06:33:36.717ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_97ec7cac-82ac-4161-b909-856eea5d7741/crypt/zone file = sled-storage/src/dataset.rs:292
3862024-09-06T06:33:36.901ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_97ec7cac-82ac-4161-b909-856eea5d7741 file = sled-storage/src/dataset.rs:378
3872024-09-06T06:33:36.909ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3882024-09-06T06:33:36.909ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3892024-09-06T06:33:36.909ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3902024-09-06T06:33:36.909ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3912024-09-06T06:33:36.909ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3922024-09-06T06:33:36.919ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3932024-09-06T06:33:36.919ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool)
3942024-09-06T06:33:37.023ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool), kind: External }
3952024-09-06T06:33:37.030ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:229
3962024-09-06T06:33:37.030ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:231
3972024-09-06T06:33:37.030ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:235
3982024-09-06T06:33:37.030ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:237
3992024-09-06T06:33:37.030ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4002024-09-06T06:33:37.030ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4012024-09-06T06:33:37.030ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
4022024-09-06T06:33:37.067ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4032024-09-06T06:33:37.075ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9d6ee5ce-b9f0-4615-ad18-41313f80d200/crypt/zone file = sled-storage/src/dataset.rs:292
4042024-09-06T06:33:37.153ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/58853ca1-8d5a-4912-87e6-57de713a602c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4052024-09-06T06:33:37.256ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9d6ee5ce-b9f0-4615-ad18-41313f80d200 file = sled-storage/src/dataset.rs:378
4062024-09-06T06:33:37.265ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4072024-09-06T06:33:37.265ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4082024-09-06T06:33:37.265ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4092024-09-06T06:33:37.265ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4102024-09-06T06:33:37.265ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4112024-09-06T06:33:37.274ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4122024-09-06T06:33:37.274ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool)
4132024-09-06T06:33:37.370ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool), kind: External }
4142024-09-06T06:33:37.377ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:229
4152024-09-06T06:33:37.378ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:231
4162024-09-06T06:33:37.378ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:235
4172024-09-06T06:33:37.378ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:237
4182024-09-06T06:33:37.378ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4192024-09-06T06:33:37.378ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4202024-09-06T06:33:37.378ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
4212024-09-06T06:33:37.412ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4222024-09-06T06:33:37.420ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a433fad0-5ca4-4fce-ac22-ce488e17cfe3/crypt/zone file = sled-storage/src/dataset.rs:292
4232024-09-06T06:33:37.597ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a433fad0-5ca4-4fce-ac22-ce488e17cfe3 file = sled-storage/src/dataset.rs:378
4242024-09-06T06:33:37.605ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4252024-09-06T06:33:37.605ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4262024-09-06T06:33:37.605ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4272024-09-06T06:33:37.605ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: acfe4e3e-8cb9-4336-9553-a7677bbf6061 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bbad1fce-0f63-461f-8b49-dd0c7a492808 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4282024-09-06T06:33:37.605ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4292024-09-06T06:33:37.605ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4302024-09-06T06:33:37.605ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4312024-09-06T06:33:37.605ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:862 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4322024-09-06T06:33:37.605ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4332024-09-06T06:33:37.605ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:888 generation = Generation(1) sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4342024-09-06T06:33:37.605ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:893 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4352024-09-06T06:33:37.605ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:898 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4362024-09-06T06:33:37.605ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4372024-09-06T06:33:37.605ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:904 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4382024-09-06T06:33:37.605ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:803 gen = Generation(1)
4392024-09-06T06:33:37.605ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:909 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4402024-09-06T06:33:37.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4110173 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48098 req_id = b67f3345-4030-4747-864f-e2186a3de4f1 response_code = 200 uri = /omicron-physical-disks
4412024-09-06T06:33:37.637ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: fe8d4251-5504-4c01-8b37-213e4ea96686, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: fc81e16c-3dde-492a-92cf-54e04297446f, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4422024-09-06T06:33:37.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61849
4432024-09-06T06:33:37.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4442024-09-06T06:33:37.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }
4452024-09-06T06:33:37.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4462024-09-06T06:33:37.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }
4472024-09-06T06:33:37.737ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4482024-09-06T06:33:37.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4492024-09-06T06:33:37.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4502024-09-06T06:33:37.786ZINFOSledAgent (ServiceManager): No ledger in /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-zones.json file = common/src/ledger.rs:188
4512024-09-06T06:33:37.786ZINFOSledAgent (ServiceManager): No ledger in /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-zones.json file = common/src/ledger.rs:188
4522024-09-06T06:33:37.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4532024-09-06T06:33:37.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4542024-09-06T06:33:37.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4552024-09-06T06:33:37.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4562024-09-06T06:33:37.854ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 19.808836257s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4572024-09-06T06:33:37.881ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556 file = illumos-utils/src/zone.rs:316
4582024-09-06T06:33:37.885ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686 file = illumos-utils/src/zone.rs:316
4592024-09-06T06:33:37.888ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f file = illumos-utils/src/zone.rs:316
4602024-09-06T06:33:37.901ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556 file = illumos-utils/src/zone.rs:354
4612024-09-06T06:33:37.905ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686 file = illumos-utils/src/zone.rs:354
4622024-09-06T06:33:37.909ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f file = illumos-utils/src/zone.rs:354
4632024-09-06T06:33:38.155ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/58853ca1-8d5a-4912-87e6-57de713a602c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
4642024-09-06T06:33:39.397ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4652024-09-06T06:33:39.397ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4662024-09-06T06:33:39.397ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
4672024-09-06T06:33:39.879ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4682024-09-06T06:33:40.357ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4692024-09-06T06:33:40.357ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4702024-09-06T06:33:40.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 65.415133ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
4712024-09-06T06:33:40.364ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4722024-09-06T06:33:40.364ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4732024-09-06T06:33:40.368ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4742024-09-06T06:33:40.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 93.278693ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
4752024-09-06T06:33:40.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 32.393862ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4762024-09-06T06:33:40.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 51.543332ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4772024-09-06T06:33:40.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 51.85327ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4782024-09-06T06:33:40.541ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 150.641661ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
4792024-09-06T06:33:40.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 90.876462ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4802024-09-06T06:33:40.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 296.640651ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4812024-09-06T06:33:40.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 220.238762ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4822024-09-06T06:33:40.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 418.3101ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
4832024-09-06T06:33:41.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 261.359206ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4842024-09-06T06:33:41.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 318.405614ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4852024-09-06T06:33:41.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 801.96776ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
4862024-09-06T06:33:41.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 458.32453ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4872024-09-06T06:33:41.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 507.46345ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4882024-09-06T06:33:41.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 1.20892935s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4892024-09-06T06:33:41.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 705.49638ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4902024-09-06T06:33:42.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 1.229256776s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
4912024-09-06T06:33:42.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 961.275812ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
4922024-09-06T06:33:42.935ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4187
4932024-09-06T06:33:42.935ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4197 gateway = fd00:1122:3344:101::1
4942024-09-06T06:33:42.942ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4276
4952024-09-06T06:33:42.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 646.216688ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
4962024-09-06T06:33:43.065ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4324
4972024-09-06T06:33:43.065ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4351
4982024-09-06T06:33:43.102ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4364
4992024-09-06T06:33:43.102ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4228
5002024-09-06T06:33:43.157ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4270
5012024-09-06T06:33:43.157ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4331 rack_subnet = fd00:1122:3344::
5022024-09-06T06:33:43.176ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4342
5032024-09-06T06:33:43.176ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4384
5042024-09-06T06:33:43.264ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4416
5052024-09-06T06:33:43.265ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4422
5062024-09-06T06:33:43.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 1.160840732s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
5072024-09-06T06:33:43.374ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4463
5082024-09-06T06:33:43.374ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5092024-09-06T06:33:43.407ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/58853ca1-8d5a-4912-87e6-57de713a602c): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:571 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
5102024-09-06T06:33:43.407ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 53.630199ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5112024-09-06T06:33:43.462ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 113.678186ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5122024-09-06T06:33:43.577ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 176.482209ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5132024-09-06T06:33:43.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 600.327455ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
5142024-09-06T06:33:43.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 605.467073ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
5152024-09-06T06:33:43.756ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5162024-09-06T06:33:43.782ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:49915
5172024-09-06T06:33:43.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:49915 req_id = e4b63bde-7dc5-4eb8-9044-cbba91b2c333 response_code = 200 uri = /sled-identifiers
5182024-09-06T06:33:43.790ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5192024-09-06T06:33:43.790ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5202024-09-06T06:33:43.790ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5212024-09-06T06:33:43.790ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5222024-09-06T06:33:43.828ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
5232024-09-06T06:33:43.877ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5242024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5252024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5262024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5272024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5282024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5292024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5302024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5312024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5322024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5332024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5342024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5352024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5362024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5372024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5382024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5392024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5402024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5412024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5422024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5432024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5442024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5452024-09-06T06:33:43.877ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5462024-09-06T06:33:44.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556") failed: Property not found. retry in 1.148816439s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
5472024-09-06T06:33:44.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 789.403224ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
5482024-09-06T06:33:44.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f") failed: Property not found. retry in 983.607965ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
5492024-09-06T06:33:45.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686") failed: Property not found. retry in 657.210037ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
5502024-09-06T06:33:45.736ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_64f7a9b7-257d-4c8d-8809-4495fa9ec556
5512024-09-06T06:33:45.736ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_fe8d4251-5504-4c01-8b37-213e4ea96686
5522024-09-06T06:33:45.736ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_fc81e16c-3dde-492a-92cf-54e04297446f
5532024-09-06T06:33:45.736ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5542024-09-06T06:33:45.737ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5552024-09-06T06:33:45.737ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8099862 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:61849 req_id = 61b60f7e-ddb5-47d6-b8cf-11163dfa6266 response_code = 204 uri = /omicron-zones
5562024-09-06T06:33:45.739ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5572024-09-06T06:33:45.878ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 131.919784ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:595
5582024-09-06T06:33:46.028ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5592024-09-06T06:33:46.081ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5602024-09-06T06:33:46.127ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5612024-09-06T06:33:46.127ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5622024-09-06T06:33:46.159ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5632024-09-06T06:33:46.161ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5642024-09-06T06:33:46.161ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5652024-09-06T06:33:46.191ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: fe8d4251-5504-4c01-8b37-213e4ea96686, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: fc81e16c-3dde-492a-92cf-54e04297446f, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: cd10d7be-1f4e-4451-aa4e-563b9db74af1, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 9fc70494-8a60-4d54-a22b-ebddff92487f, kind: Service { id: cd10d7be-1f4e-4451-aa4e-563b9db74af1 }, name: Name("ntp-cd10d7be-1f4e-4451-aa4e-563b9db74af1"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5662024-09-06T06:33:46.191ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40032
5672024-09-06T06:33:46.191ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5682024-09-06T06:33:46.191ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }
5692024-09-06T06:33:46.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5702024-09-06T06:33:46.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }
5712024-09-06T06:33:46.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5722024-09-06T06:33:46.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5732024-09-06T06:33:46.248ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5742024-09-06T06:33:46.248ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-zones.json file = common/src/ledger.rs:177
5752024-09-06T06:33:46.248ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-zones.json file = common/src/ledger.rs:177
5762024-09-06T06:33:46.248ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5772024-09-06T06:33:46.259ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5782024-09-06T06:33:46.291ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5792024-09-06T06:33:46.293ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5802024-09-06T06:33:46.293ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5812024-09-06T06:33:46.324ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 168, 161]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5822024-09-06T06:33:46.324ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = boundary_ntp
5832024-09-06T06:33:46.324ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5842024-09-06T06:33:46.324ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5852024-09-06T06:33:46.325ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5862024-09-06T06:33:46.326ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5872024-09-06T06:33:46.349ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1 file = illumos-utils/src/zone.rs:316
5882024-09-06T06:33:46.369ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1 file = illumos-utils/src/zone.rs:354
5892024-09-06T06:33:46.988ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5902024-09-06T06:33:46.988ZINFOSledAgent (ServiceManager): Booting oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5912024-09-06T06:33:47.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 62.095317ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5922024-09-06T06:33:47.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 99.084025ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5932024-09-06T06:33:47.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 291.294211ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5942024-09-06T06:33:47.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 244.68732ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5952024-09-06T06:33:47.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 1.139375623s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5962024-09-06T06:33:49.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 884.054267ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5972024-09-06T06:33:49.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 650.015815ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5982024-09-06T06:33:50.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 956.395239ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
5992024-09-06T06:33:51.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1") failed: Property not found. retry in 1.25616329s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
6002024-09-06T06:33:52.819ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1
6012024-09-06T06:33:52.820ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6022024-09-06T06:33:52.820ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6032024-09-06T06:33:52.820ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6629245 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:40032 req_id = 77a4b65f-524e-42e8-bc00-243f772dec41 response_code = 204 uri = /omicron-zones
6042024-09-06T06:33:52.823ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6052024-09-06T06:33:52.853ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6062024-09-06T06:33:52.853ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46005
6072024-09-06T06:33:52.861ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6082024-09-06T06:33:52.861ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8077 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46005 req_id = 0df12f06-d66a-43c4-9953-0356bce73d6c response_code = 500 uri = /timesync
6092024-09-06T06:33:52.864ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6102024-09-06T06:33:53.035ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6112024-09-06T06:33:53.036ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36929
6122024-09-06T06:33:57.045ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6132024-09-06T06:33:57.045ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4009580 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36929 req_id = da018e9f-661c-4a57-a3a2-5d6ee4cfeeca response_code = 500 uri = /timesync
6142024-09-06T06:33:57.048ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6152024-09-06T06:33:57.253ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6162024-09-06T06:33:57.253ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56594
6172024-09-06T06:33:57.262ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6182024-09-06T06:33:57.262ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8347 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56594 req_id = 0b53ae0c-aaf5-4ce7-8620-acb0d210c85c response_code = 500 uri = /timesync
6192024-09-06T06:33:57.264ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6202024-09-06T06:33:57.400ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6212024-09-06T06:33:57.400ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54539
6222024-09-06T06:34:02.057ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_cd10d7be-1f4e-4451-aa4e-563b9db74af1': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6232024-09-06T06:34:02.058ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4008932 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54539 req_id = cf68e939-5c38-4a58-8c2f-533378e1054f response_code = 500 uri = /timesync
6242024-09-06T06:34:02.061ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6252024-09-06T06:34:02.321ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6262024-09-06T06:34:02.321ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49394
6272024-09-06T06:34:02.330ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8840 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49394 req_id = 1d2571d7-0942-4718-bc93-b9f01ccb7e0c response_code = 200 uri = /timesync
6282024-09-06T06:34:02.332ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.09241093 } file = sled-agent/src/rack_setup/service.rs:660
6292024-09-06T06:34:02.332ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6302024-09-06T06:34:02.605ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6312024-09-06T06:34:02.605ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57688
6322024-09-06T06:34:02.613ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57688 req_id = 0224587d-58e1-4992-93c3-f33902c5ca5a response_code = 200 uri = /timesync
6332024-09-06T06:34:02.615ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.091644347 } file = sled-agent/src/rack_setup/service.rs:660
6342024-09-06T06:34:02.615ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6352024-09-06T06:34:02.958ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6362024-09-06T06:34:02.958ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47657
6372024-09-06T06:34:02.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8590 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47657 req_id = dc09c315-59d3-4ade-925f-445604d46020 response_code = 200 uri = /timesync
6382024-09-06T06:34:02.968ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.089215122 } file = sled-agent/src/rack_setup/service.rs:660
6392024-09-06T06:34:02.968ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6402024-09-06T06:34:03.440ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6412024-09-06T06:34:03.440ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42263
6422024-09-06T06:34:03.448ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42263 req_id = 0f67ac4d-a79f-4438-8c6e-da32fa552d9d response_code = 200 uri = /timesync
6432024-09-06T06:34:03.449ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.090393685 } file = sled-agent/src/rack_setup/service.rs:660
6442024-09-06T06:34:03.449ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6452024-09-06T06:34:03.864ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6462024-09-06T06:34:03.864ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34478
6472024-09-06T06:34:03.873ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9241 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34478 req_id = 40fb481d-0a8c-4f3a-9087-7d4c1a9f1074 response_code = 200 uri = /timesync
6482024-09-06T06:34:03.875ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.089244835 } file = sled-agent/src/rack_setup/service.rs:660
6492024-09-06T06:34:03.875ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6502024-09-06T06:34:04.372ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6512024-09-06T06:34:04.372ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58572
6522024-09-06T06:34:04.380ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8040 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58572 req_id = 70a6cb4e-4ce1-4cf1-943b-d5f230c2af56 response_code = 200 uri = /timesync
6532024-09-06T06:34:04.381ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.087116972 } file = sled-agent/src/rack_setup/service.rs:660
6542024-09-06T06:34:04.381ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6552024-09-06T06:34:04.842ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6562024-09-06T06:34:04.842ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47345
6572024-09-06T06:34:04.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8282 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47345 req_id = cdc8b9aa-7924-4c15-ad40-a14034aeb5c5 response_code = 200 uri = /timesync
6582024-09-06T06:34:04.853ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.085845396 } file = sled-agent/src/rack_setup/service.rs:660
6592024-09-06T06:34:04.853ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6602024-09-06T06:34:05.735ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6612024-09-06T06:34:05.735ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60866
6622024-09-06T06:34:05.744ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8811 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60866 req_id = 636c1564-a192-451b-8a2a-23f71fd78b4a response_code = 200 uri = /timesync
6632024-09-06T06:34:05.745ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1725604440.449202, correction: 0.086358063 } file = sled-agent/src/rack_setup/service.rs:660
6642024-09-06T06:34:05.745ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6652024-09-06T06:34:06.701ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6662024-09-06T06:34:06.701ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62589
6672024-09-06T06:34:06.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9529 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62589 req_id = 093f621a-5193-4135-ac37-cacd1e23ecea response_code = 200 uri = /timesync
6682024-09-06T06:34:06.712ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604445.788746, correction: 0.084062383 } file = sled-agent/src/rack_setup/service.rs:660
6692024-09-06T06:34:06.712ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6702024-09-06T06:34:07.642ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6712024-09-06T06:34:07.642ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38123
6722024-09-06T06:34:07.651ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8615 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38123 req_id = 6bfa0a9d-4057-484f-9aed-6965499b7c66 response_code = 200 uri = /timesync
6732024-09-06T06:34:07.654ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604447.0945425, correction: 0.081760958 } file = sled-agent/src/rack_setup/service.rs:660
6742024-09-06T06:34:07.654ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6752024-09-06T06:34:08.891ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6762024-09-06T06:34:08.891ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43432
6772024-09-06T06:34:08.900ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8618 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43432 req_id = 38295086-a4e7-4e8c-8974-beafcfb042be response_code = 200 uri = /timesync
6782024-09-06T06:34:08.901ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604447.8802366, correction: 0.078777939 } file = sled-agent/src/rack_setup/service.rs:660
6792024-09-06T06:34:08.901ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6802024-09-06T06:34:10.259ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6812024-09-06T06:34:10.259ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58091
6822024-09-06T06:34:10.267ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8282 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58091 req_id = d31e3b92-d6ee-4a98-b252-516b65adc925 response_code = 200 uri = /timesync
6832024-09-06T06:34:10.269ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604448.9084964, correction: 0.074204192 } file = sled-agent/src/rack_setup/service.rs:660
6842024-09-06T06:34:10.270ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6852024-09-06T06:34:11.123ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6862024-09-06T06:34:11.123ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54319
6872024-09-06T06:34:11.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7563 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54319 req_id = 8f5f7dcc-2467-4a8d-b0a2-381f3bc9a505 response_code = 200 uri = /timesync
6882024-09-06T06:34:11.132ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604450.0512593, correction: 0.071449921 } file = sled-agent/src/rack_setup/service.rs:660
6892024-09-06T06:34:11.132ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6902024-09-06T06:34:13.792ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6912024-09-06T06:34:13.792ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59418
6922024-09-06T06:34:13.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9232 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59418 req_id = f433102d-652c-4e82-96b3-0bb466be3dd5 response_code = 200 uri = /timesync
6932024-09-06T06:34:13.803ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604452.670814, correction: 0.063712083 } file = sled-agent/src/rack_setup/service.rs:660
6942024-09-06T06:34:13.803ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6952024-09-06T06:34:13.832ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60653
6962024-09-06T06:34:13.832ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60653 req_id = 3e82b071-c654-4226-852c-394e187bd0d2 response_code = 200 uri = /baseboard
6972024-09-06T06:34:16.167ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6982024-09-06T06:34:16.167ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59409
6992024-09-06T06:34:16.177ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9327 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59409 req_id = eb2ef9a8-0c5b-42dc-a86c-e2dd185f1be9 response_code = 200 uri = /timesync
7002024-09-06T06:34:16.179ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604454.586499, correction: 0.057398677 } file = sled-agent/src/rack_setup/service.rs:660
7012024-09-06T06:34:16.179ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7022024-09-06T06:34:17.137ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 150.579822ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
7032024-09-06T06:34:17.690ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7042024-09-06T06:34:17.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65268
7052024-09-06T06:34:17.699ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8774 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65268 req_id = 6d10eca2-44ae-436d-b822-ba8e39b55bb3 response_code = 200 uri = /timesync
7062024-09-06T06:34:17.701ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604456.6507268, correction: 0.053668704 } file = sled-agent/src/rack_setup/service.rs:660
7072024-09-06T06:34:17.701ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7082024-09-06T06:34:19.572ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7092024-09-06T06:34:19.572ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43946
7102024-09-06T06:34:19.594ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 21542 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43946 req_id = 67dfd194-b709-41da-93a3-4e40782ae276 response_code = 200 uri = /timesync
7112024-09-06T06:34:19.596ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 1209934681, ip_addr: 72.30.35.89, stratum: 3, ref_time: 1725604458.5650876, correction: 0.048246909 } file = sled-agent/src/rack_setup/service.rs:660
7122024-09-06T06:34:19.596ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1299
7132024-09-06T06:34:19.628ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: fe8d4251-5504-4c01-8b37-213e4ea96686, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: fc81e16c-3dde-492a-92cf-54e04297446f, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: cd10d7be-1f4e-4451-aa4e-563b9db74af1, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 9fc70494-8a60-4d54-a22b-ebddff92487f, kind: Service { id: cd10d7be-1f4e-4451-aa4e-563b9db74af1 }, name: Name("ntp-cd10d7be-1f4e-4451-aa4e-563b9db74af1"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 797dee39-1d18-46b4-84d2-ec1ac80da063, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } } } }, OmicronZoneConfig { id: 78f74ead-aba1-47a6-9d71-5457e9371876, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } } } }, OmicronZoneConfig { id: e36becb1-b59c-4133-9ed3-40d576551ca6, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } } } }, OmicronZoneConfig { id: f967676b-eab7-4b42-aa18-33fce764e53b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0815d824-8ac0-449b-b634-c08e09ddf5ba, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7142024-09-06T06:34:19.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53498
7152024-09-06T06:34:19.629ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7162024-09-06T06:34:19.629ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }
7172024-09-06T06:34:19.650ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7182024-09-06T06:34:19.650ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }
7192024-09-06T06:34:19.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7202024-09-06T06:34:19.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7212024-09-06T06:34:19.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 797dee39-1d18-46b4-84d2-ec1ac80da063, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7222024-09-06T06:34:19.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 797dee39-1d18-46b4-84d2-ec1ac80da063, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Cockroach }, responder: ... }
7232024-09-06T06:34:19.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 78f74ead-aba1-47a6-9d71-5457e9371876, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7242024-09-06T06:34:19.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 78f74ead-aba1-47a6-9d71-5457e9371876, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: Cockroach }, responder: ... }
7252024-09-06T06:34:19.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e36becb1-b59c-4133-9ed3-40d576551ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7262024-09-06T06:34:19.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e36becb1-b59c-4133-9ed3-40d576551ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7272024-09-06T06:34:19.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f967676b-eab7-4b42-aa18-33fce764e53b, dataset_name: DatasetName { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7282024-09-06T06:34:19.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f967676b-eab7-4b42-aa18-33fce764e53b, dataset_name: DatasetName { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7292024-09-06T06:34:19.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0815d824-8ac0-449b-b634-c08e09ddf5ba, dataset_name: DatasetName { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7302024-09-06T06:34:19.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0815d824-8ac0-449b-b634-c08e09ddf5ba, dataset_name: DatasetName { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7312024-09-06T06:34:19.956ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7322024-09-06T06:34:19.956ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-zones.json file = common/src/ledger.rs:177
7332024-09-06T06:34:19.956ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-zones.json file = common/src/ledger.rs:177
7342024-09-06T06:34:19.956ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7352024-09-06T06:34:20.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7362024-09-06T06:34:20.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7372024-09-06T06:34:20.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7382024-09-06T06:34:20.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7392024-09-06T06:34:20.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7402024-09-06T06:34:20.132ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876 file = illumos-utils/src/zone.rs:316
7412024-09-06T06:34:20.136ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6 file = illumos-utils/src/zone.rs:316
7422024-09-06T06:34:20.141ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b file = illumos-utils/src/zone.rs:316
7432024-09-06T06:34:20.144ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063 file = illumos-utils/src/zone.rs:316
7442024-09-06T06:34:20.149ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba file = illumos-utils/src/zone.rs:316
7452024-09-06T06:34:20.154ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876 file = illumos-utils/src/zone.rs:354
7462024-09-06T06:34:20.159ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6 file = illumos-utils/src/zone.rs:354
7472024-09-06T06:34:20.164ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b file = illumos-utils/src/zone.rs:354
7482024-09-06T06:34:20.169ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063 file = illumos-utils/src/zone.rs:354
7492024-09-06T06:34:20.173ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba file = illumos-utils/src/zone.rs:354
7502024-09-06T06:34:23.146ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='78f74ead-aba1-47a6-9d71-5457e9371876'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7512024-09-06T06:34:23.147ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7522024-09-06T06:34:23.154ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='797dee39-1d18-46b4-84d2-ec1ac80da063'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7532024-09-06T06:34:23.154ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7542024-09-06T06:34:23.164ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='0815d824-8ac0-449b-b634-c08e09ddf5ba'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7552024-09-06T06:34:23.164ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e36becb1-b59c-4133-9ed3-40d576551ca6'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7562024-09-06T06:34:23.165ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7572024-09-06T06:34:23.170ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7582024-09-06T06:34:23.176ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='f967676b-eab7-4b42-aa18-33fce764e53b'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7592024-09-06T06:34:23.177ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7602024-09-06T06:34:23.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 40.328095ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7612024-09-06T06:34:23.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 30.654281ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7622024-09-06T06:34:23.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 25.022864ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7632024-09-06T06:34:23.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 67.963998ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7642024-09-06T06:34:23.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 67.988433ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7652024-09-06T06:34:23.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 100.797265ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7662024-09-06T06:34:23.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 148.650452ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7672024-09-06T06:34:23.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 89.456342ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7682024-09-06T06:34:23.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 95.813689ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7692024-09-06T06:34:23.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 112.645626ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7702024-09-06T06:34:23.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 189.732334ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7712024-09-06T06:34:24.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 273.123959ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7722024-09-06T06:34:24.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 214.461603ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7732024-09-06T06:34:24.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 110.903575ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7742024-09-06T06:34:24.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 347.880144ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7752024-09-06T06:34:24.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 228.543602ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7762024-09-06T06:34:24.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 136.984885ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7772024-09-06T06:34:25.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 368.174727ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7782024-09-06T06:34:25.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 288.959479ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7792024-09-06T06:34:25.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 505.760673ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7802024-09-06T06:34:26.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 685.905401ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7812024-09-06T06:34:26.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 249.026902ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7822024-09-06T06:34:26.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 1.19573611s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7832024-09-06T06:34:26.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 579.843028ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7842024-09-06T06:34:26.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 1.086060876s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7852024-09-06T06:34:26.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 514.827664ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7862024-09-06T06:34:26.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 771.884555ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7872024-09-06T06:34:27.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 811.972793ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7882024-09-06T06:34:27.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 1.087149133s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7892024-09-06T06:34:27.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 1.442092144s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
7902024-09-06T06:34:27.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 581.638359ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7912024-09-06T06:34:27.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 838.124632ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7922024-09-06T06:34:27.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 1.02912426s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7932024-09-06T06:34:28.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 723.525464ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7942024-09-06T06:34:28.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 523.55235ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7952024-09-06T06:34:28.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 1.47057715s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
7962024-09-06T06:34:28.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.470964566s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
7972024-09-06T06:34:28.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 793.338346ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
7982024-09-06T06:34:28.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 1.200171106s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
7992024-09-06T06:34:29.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 1.440584611s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
8002024-09-06T06:34:29.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 658.98968ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
8012024-09-06T06:34:30.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 823.204951ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
8022024-09-06T06:34:30.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 591.782889ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
8032024-09-06T06:34:30.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 640.492804ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8042024-09-06T06:34:30.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 1.093425662s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
8052024-09-06T06:34:30.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 855.787016ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
8062024-09-06T06:34:30.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 1.009465655s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
8072024-09-06T06:34:30.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 1.052040501s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
8082024-09-06T06:34:30.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.390813366s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8092024-09-06T06:34:31.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 1.129722484s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
8102024-09-06T06:34:31.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 662.590593ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
8112024-09-06T06:34:31.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 1.425601084s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
8122024-09-06T06:34:31.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 631.358835ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
8132024-09-06T06:34:32.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 1.094093158s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
8142024-09-06T06:34:32.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.09624454s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8152024-09-06T06:34:32.363ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 536.559474ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
8162024-09-06T06:34:32.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 695.492704ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
8172024-09-06T06:34:32.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 1.191817797s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
8182024-09-06T06:34:33.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b") failed: Property not found. retry in 1.314338209s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
8192024-09-06T06:34:33.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876") failed: Property not found. retry in 994.887608ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
8202024-09-06T06:34:33.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba") failed: Property not found. retry in 1.070124115s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
8212024-09-06T06:34:33.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 549.500203ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8222024-09-06T06:34:33.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063") failed: Property not found. retry in 1.32170673s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
8232024-09-06T06:34:33.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.005704231s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8242024-09-06T06:34:34.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.091871613s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8252024-09-06T06:34:36.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.326504677s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8262024-09-06T06:34:37.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.44324714s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8272024-09-06T06:34:38.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.090452465s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8282024-09-06T06:34:39.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.24132193s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8292024-09-06T06:34:41.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.219562929s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8302024-09-06T06:34:42.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 944.333337ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8312024-09-06T06:34:43.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6") failed: Property not found. retry in 1.253126277s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8322024-09-06T06:34:43.941ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34659
8332024-09-06T06:34:43.941ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34659 req_id = 9a0e247c-af8b-4ff7-b1c1-fdac957da632 response_code = 200 uri = /baseboard
8342024-09-06T06:34:44.693ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_78f74ead-aba1-47a6-9d71-5457e9371876
8352024-09-06T06:34:44.693ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_e36becb1-b59c-4133-9ed3-40d576551ca6
8362024-09-06T06:34:44.693ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_f967676b-eab7-4b42-aa18-33fce764e53b
8372024-09-06T06:34:44.693ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_797dee39-1d18-46b4-84d2-ec1ac80da063
8382024-09-06T06:34:44.693ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_0815d824-8ac0-449b-b634-c08e09ddf5ba
8392024-09-06T06:34:44.693ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8402024-09-06T06:34:44.745ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8412024-09-06T06:34:44.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25215440 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53498 req_id = dac39225-4a61-4613-b0f1-c934df77db0a response_code = 204 uri = /omicron-zones
8422024-09-06T06:34:44.937ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54043
8432024-09-06T06:34:44.967ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::7]:32221 file = sled-agent/src/services.rs:3662
8442024-09-06T06:34:47.958ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 614.79184ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
8452024-09-06T06:34:55.696ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3680
8462024-09-06T06:35:03.494ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3703
8472024-09-06T06:35:03.494ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 18566833 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:54043 req_id = e172a9ae-61c5-4329-b9b8-5ff68f33a6b0 response_code = 204 uri = /cockroachdb
8482024-09-06T06:35:03.528ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: fe8d4251-5504-4c01-8b37-213e4ea96686, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: fc81e16c-3dde-492a-92cf-54e04297446f, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: cd10d7be-1f4e-4451-aa4e-563b9db74af1, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 9fc70494-8a60-4d54-a22b-ebddff92487f, kind: Service { id: cd10d7be-1f4e-4451-aa4e-563b9db74af1 }, name: Name("ntp-cd10d7be-1f4e-4451-aa4e-563b9db74af1"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 161])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 797dee39-1d18-46b4-84d2-ec1ac80da063, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } } } }, OmicronZoneConfig { id: 78f74ead-aba1-47a6-9d71-5457e9371876, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } } } }, OmicronZoneConfig { id: e36becb1-b59c-4133-9ed3-40d576551ca6, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } } } }, OmicronZoneConfig { id: f967676b-eab7-4b42-aa18-33fce764e53b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0815d824-8ac0-449b-b634-c08e09ddf5ba, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: f4c090fd-0141-407e-a547-8163f49da46c, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: e9c280ee-8313-41ab-8fa4-df346c954189, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: e919b437-ca84-4c31-970f-ae903b0601dc, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c99a3832-620a-48f7-b103-35c8b28bf618, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } } } }, OmicronZoneConfig { id: 6395a09e-9850-4d3d-bced-7fead6fd0c31, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 682aec46-4e71-4cbe-a6b3-ae4527771b66, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: e9284324-b820-4122-bd9c-c86c1a78e569, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 789dddab-4b92-4605-841e-47bdff42904a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External } } } }, OmicronZoneConfig { id: 1d333596-f29b-425b-a7df-0e33acb1d598, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External } } } }, OmicronZoneConfig { id: dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a0f50b7-0301-4b31-ab4b-e47d8d5e2421, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External } } } }, OmicronZoneConfig { id: 89028fef-e2fe-45a6-a453-d7e529ee2181, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External } } } }, OmicronZoneConfig { id: e11f7659-c40d-4b11-8887-429b1186d7b7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool), kind: External } } } }, OmicronZoneConfig { id: e90b2a1c-bdff-4e4e-b069-89ce02023950, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7ea4e3c3-4afc-46a9-9d22-0f924a266ab8, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool), kind: External } } } }, OmicronZoneConfig { id: ce5f16f3-697d-4f00-ade9-701fa0baf1a5, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8492024-09-06T06:35:03.528ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56930
8502024-09-06T06:35:03.529ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8512024-09-06T06:35:03.529ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fe8d4251-5504-4c01-8b37-213e4ea96686, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: InternalDns }, responder: ... }
8522024-09-06T06:35:03.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8532024-09-06T06:35:03.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: fc81e16c-3dde-492a-92cf-54e04297446f, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: InternalDns }, responder: ... }
8542024-09-06T06:35:03.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8552024-09-06T06:35:03.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 64f7a9b7-257d-4c8d-8809-4495fa9ec556, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8562024-09-06T06:35:03.591ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 797dee39-1d18-46b4-84d2-ec1ac80da063, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8572024-09-06T06:35:03.591ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 797dee39-1d18-46b4-84d2-ec1ac80da063, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Cockroach }, responder: ... }
8582024-09-06T06:35:03.610ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 78f74ead-aba1-47a6-9d71-5457e9371876, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8592024-09-06T06:35:03.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 78f74ead-aba1-47a6-9d71-5457e9371876, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: Cockroach }, responder: ... }
8602024-09-06T06:35:03.616ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.242627925s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
8612024-09-06T06:35:03.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e36becb1-b59c-4133-9ed3-40d576551ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8622024-09-06T06:35:03.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e36becb1-b59c-4133-9ed3-40d576551ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8632024-09-06T06:35:03.650ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f967676b-eab7-4b42-aa18-33fce764e53b, dataset_name: DatasetName { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8642024-09-06T06:35:03.650ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f967676b-eab7-4b42-aa18-33fce764e53b, dataset_name: DatasetName { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8652024-09-06T06:35:03.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0815d824-8ac0-449b-b634-c08e09ddf5ba, dataset_name: DatasetName { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8662024-09-06T06:35:03.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0815d824-8ac0-449b-b634-c08e09ddf5ba, dataset_name: DatasetName { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8672024-09-06T06:35:03.690ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81c3dd8c-a34f-489a-91c0-599e726f0b45, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8682024-09-06T06:35:03.690ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81c3dd8c-a34f-489a-91c0-599e726f0b45, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8692024-09-06T06:35:03.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8702024-09-06T06:35:03.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8712024-09-06T06:35:03.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c99a3832-620a-48f7-b103-35c8b28bf618, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
8722024-09-06T06:35:03.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c99a3832-620a-48f7-b103-35c8b28bf618, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8732024-09-06T06:35:03.978ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 789dddab-4b92-4605-841e-47bdff42904a, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8742024-09-06T06:35:03.978ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 789dddab-4b92-4605-841e-47bdff42904a, dataset_name: DatasetName { pool_name: ZpoolName { id: 7516b8f4-123a-4b07-8c44-30d87eec7e9c (zpool), kind: External }, kind: Crucible }, responder: ... }
8752024-09-06T06:35:04.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1d333596-f29b-425b-a7df-0e33acb1d598, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8762024-09-06T06:35:04.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 1d333596-f29b-425b-a7df-0e33acb1d598, dataset_name: DatasetName { pool_name: ZpoolName { id: 56647842-7219-46e1-9978-bdffa4d4e8cf (zpool), kind: External }, kind: Crucible }, responder: ... }
8772024-09-06T06:35:04.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8782024-09-06T06:35:04.158ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a42b808-7b1f-406b-87b7-e6aeecc73464 (zpool), kind: External }, kind: Crucible }, responder: ... }
8792024-09-06T06:35:04.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a0f50b7-0301-4b31-ab4b-e47d8d5e2421, dataset_name: DatasetName { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8802024-09-06T06:35:04.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a0f50b7-0301-4b31-ab4b-e47d8d5e2421, dataset_name: DatasetName { pool_name: ZpoolName { id: a32e2a10-4360-4704-b36b-dc63733b1b01 (zpool), kind: External }, kind: Crucible }, responder: ... }
8812024-09-06T06:35:04.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 89028fef-e2fe-45a6-a453-d7e529ee2181, dataset_name: DatasetName { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8822024-09-06T06:35:04.333ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 89028fef-e2fe-45a6-a453-d7e529ee2181, dataset_name: DatasetName { pool_name: ZpoolName { id: f881a8f5-60a1-4050-a3bb-fd18022091f4 (zpool), kind: External }, kind: Crucible }, responder: ... }
8832024-09-06T06:35:04.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e11f7659-c40d-4b11-8887-429b1186d7b7, dataset_name: DatasetName { pool_name: ZpoolName { id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8842024-09-06T06:35:04.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e11f7659-c40d-4b11-8887-429b1186d7b7, dataset_name: DatasetName { pool_name: ZpoolName { id: 681fcf67-79af-4c28-8e2f-62c72e00e09a (zpool), kind: External }, kind: Crucible }, responder: ... }
8852024-09-06T06:35:04.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e90b2a1c-bdff-4e4e-b069-89ce02023950, dataset_name: DatasetName { pool_name: ZpoolName { id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8862024-09-06T06:35:04.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e90b2a1c-bdff-4e4e-b069-89ce02023950, dataset_name: DatasetName { pool_name: ZpoolName { id: 97ec7cac-82ac-4161-b909-856eea5d7741 (zpool), kind: External }, kind: Crucible }, responder: ... }
8872024-09-06T06:35:04.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7ea4e3c3-4afc-46a9-9d22-0f924a266ab8, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8882024-09-06T06:35:04.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 7ea4e3c3-4afc-46a9-9d22-0f924a266ab8, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d6ee5ce-b9f0-4615-ad18-41313f80d200 (zpool), kind: External }, kind: Crucible }, responder: ... }
8892024-09-06T06:35:04.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ce5f16f3-697d-4f00-ade9-701fa0baf1a5, dataset_name: DatasetName { pool_name: ZpoolName { id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8902024-09-06T06:35:04.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ce5f16f3-697d-4f00-ade9-701fa0baf1a5, dataset_name: DatasetName { pool_name: ZpoolName { id: a433fad0-5ca4-4fce-ac22-ce488e17cfe3 (zpool), kind: External }, kind: Crucible }, responder: ... }
8912024-09-06T06:35:04.750ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8922024-09-06T06:35:04.750ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-zones.json file = common/src/ledger.rs:177
8932024-09-06T06:35:04.750ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-zones.json file = common/src/ledger.rs:177
8942024-09-06T06:35:04.750ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8952024-09-06T06:35:04.849ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
8962024-09-06T06:35:04.865ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
8972024-09-06T06:35:04.865ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8982024-09-06T06:35:04.865ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8992024-09-06T06:35:04.890ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9002024-09-06T06:35:04.890ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9012024-09-06T06:35:04.927ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9022024-09-06T06:35:04.927ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9032024-09-06T06:35:04.927ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9042024-09-06T06:35:04.927ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9052024-09-06T06:35:05.136ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9062024-09-06T06:35:05.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9072024-09-06T06:35:05.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9082024-09-06T06:35:05.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9092024-09-06T06:35:05.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9102024-09-06T06:35:05.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9112024-09-06T06:35:05.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9122024-09-06T06:35:05.150ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9132024-09-06T06:35:05.150ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9142024-09-06T06:35:05.150ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618 file = illumos-utils/src/zone.rs:316
9152024-09-06T06:35:05.156ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569 file = illumos-utils/src/zone.rs:316
9162024-09-06T06:35:05.160ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8 file = illumos-utils/src/zone.rs:316
9172024-09-06T06:35:05.165ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc file = illumos-utils/src/zone.rs:316
9182024-09-06T06:35:05.170ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24 file = illumos-utils/src/zone.rs:316
9192024-09-06T06:35:05.176ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a file = illumos-utils/src/zone.rs:316
9202024-09-06T06:35:05.181ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950 file = illumos-utils/src/zone.rs:316
9212024-09-06T06:35:05.324ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618 file = illumos-utils/src/zone.rs:354
9222024-09-06T06:35:05.329ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569 file = illumos-utils/src/zone.rs:354
9232024-09-06T06:35:05.334ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8 file = illumos-utils/src/zone.rs:354
9242024-09-06T06:35:05.338ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc file = illumos-utils/src/zone.rs:354
9252024-09-06T06:35:05.343ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24 file = illumos-utils/src/zone.rs:354
9262024-09-06T06:35:05.348ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a file = illumos-utils/src/zone.rs:354
9272024-09-06T06:35:05.353ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950 file = illumos-utils/src/zone.rs:354
9282024-09-06T06:35:05.358ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31 file = illumos-utils/src/zone.rs:316
9292024-09-06T06:35:05.363ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421 file = illumos-utils/src/zone.rs:316
9302024-09-06T06:35:05.369ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598 file = illumos-utils/src/zone.rs:316
9312024-09-06T06:35:05.374ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181 file = illumos-utils/src/zone.rs:316
9322024-09-06T06:35:05.379ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7 file = illumos-utils/src/zone.rs:316
9332024-09-06T06:35:05.384ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5 file = illumos-utils/src/zone.rs:316
9342024-09-06T06:35:05.390ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66 file = illumos-utils/src/zone.rs:316
9352024-09-06T06:35:05.397ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31 file = illumos-utils/src/zone.rs:354
9362024-09-06T06:35:05.404ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421 file = illumos-utils/src/zone.rs:354
9372024-09-06T06:35:05.410ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598 file = illumos-utils/src/zone.rs:354
9382024-09-06T06:35:05.415ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181 file = illumos-utils/src/zone.rs:354
9392024-09-06T06:35:05.420ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7 file = illumos-utils/src/zone.rs:354
9402024-09-06T06:35:05.425ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5 file = illumos-utils/src/zone.rs:354
9412024-09-06T06:35:05.430ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66 file = illumos-utils/src/zone.rs:354
9422024-09-06T06:35:05.479ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9432024-09-06T06:35:05.530ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9442024-09-06T06:35:05.594ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9452024-09-06T06:35:05.639ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9462024-09-06T06:35:05.687ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9472024-09-06T06:35:05.689ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9482024-09-06T06:35:05.689ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9492024-09-06T06:35:05.739ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 252, 19]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9502024-09-06T06:35:05.739ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
9512024-09-06T06:35:05.740ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9522024-09-06T06:35:05.741ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9532024-09-06T06:35:05.793ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 139, 148]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9542024-09-06T06:35:05.793ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
9552024-09-06T06:35:05.795ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9562024-09-06T06:35:05.795ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9572024-09-06T06:35:05.847ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 214, 255]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
9582024-09-06T06:35:05.847ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
9592024-09-06T06:35:05.849ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9602024-09-06T06:35:05.849ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9612024-09-06T06:35:05.901ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 176, 154]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9622024-09-06T06:35:05.901ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
9632024-09-06T06:35:05.902ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9642024-09-06T06:35:05.903ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9652024-09-06T06:35:05.958ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 181, 227]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
9662024-09-06T06:35:05.958ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
9672024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
9682024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
9692024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
9702024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
9712024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
9722024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
9732024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
9742024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
9752024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
9762024-09-06T06:35:05.961ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
9772024-09-06T06:35:05.962ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
9782024-09-06T06:35:05.964ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9792024-09-06T06:35:06.010ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
9802024-09-06T06:35:06.015ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
9812024-09-06T06:35:06.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9822024-09-06T06:35:06.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9832024-09-06T06:35:06.019ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
9842024-09-06T06:35:06.023ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
9852024-09-06T06:35:06.026ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9862024-09-06T06:35:06.026ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9872024-09-06T06:35:06.282ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45 file = illumos-utils/src/zone.rs:316
9882024-09-06T06:35:06.302ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c file = illumos-utils/src/zone.rs:316
9892024-09-06T06:35:06.319ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189 file = illumos-utils/src/zone.rs:316
9902024-09-06T06:35:06.347ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d file = illumos-utils/src/zone.rs:316
9912024-09-06T06:35:06.381ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476 file = illumos-utils/src/zone.rs:316
9922024-09-06T06:35:06.404ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45 file = illumos-utils/src/zone.rs:354
9932024-09-06T06:35:06.418ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c file = illumos-utils/src/zone.rs:354
9942024-09-06T06:35:06.429ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189 file = illumos-utils/src/zone.rs:354
9952024-09-06T06:35:06.438ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d file = illumos-utils/src/zone.rs:354
9962024-09-06T06:35:06.447ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476 file = illumos-utils/src/zone.rs:354
9972024-09-06T06:35:07.493ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9982024-09-06T06:35:07.493ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
9992024-09-06T06:35:07.543ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7516b8f4-123a-4b07-8c44-30d87eec7e9c/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3530f58b-786b-4385-869f-e5a4674d7c84'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10002024-09-06T06:35:07.543ZINFOSledAgent (ServiceManager): Booting oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
10012024-09-06T06:35:07.588ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_97ec7cac-82ac-4161-b909-856eea5d7741/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a3642754-5b86-4c49-a142-8db95fd6dda7'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10022024-09-06T06:35:07.588ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_9d6ee5ce-b9f0-4615-ad18-41313f80d200/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='819bf4f5-f44f-43b3-a202-fec57809ec29'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10032024-09-06T06:35:07.588ZINFOSledAgent (ServiceManager): Booting oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
10042024-09-06T06:35:07.594ZINFOSledAgent (ServiceManager): Booting oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10052024-09-06T06:35:07.719ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='e919b437-ca84-4c31-970f-ae903b0601dc'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10062024-09-06T06:35:07.720ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
10072024-09-06T06:35:07.755ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10082024-09-06T06:35:07.755ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
10092024-09-06T06:35:07.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 73.84649ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
10102024-09-06T06:35:07.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 38.8131ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10112024-09-06T06:35:07.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 34.186982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
10122024-09-06T06:35:07.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 51.563244ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
10132024-09-06T06:35:07.836ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_681fcf67-79af-4c28-8e2f-62c72e00e09a/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='4fbb189f-5c3b-4ae0-b70e-4381df0f31f6'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10142024-09-06T06:35:07.836ZINFOSledAgent (ServiceManager): Booting oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
10152024-09-06T06:35:07.843ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a32e2a10-4360-4704-b36b-dc63733b1b01/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='13016e03-6190-4ddf-95cb-f47e41247209'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10162024-09-06T06:35:07.843ZINFOSledAgent (ServiceManager): Booting oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
10172024-09-06T06:35:07.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 113.357234ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10182024-09-06T06:35:07.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 85.039856ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
10192024-09-06T06:35:07.878ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a433fad0-5ca4-4fce-ac22-ce488e17cfe3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='27e782e2-6ad1-4602-9973-c5bd5e75ff06'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10202024-09-06T06:35:07.878ZINFOSledAgent (ServiceManager): Booting oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
10212024-09-06T06:35:08.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 142.837101ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
10222024-09-06T06:35:08.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 174.142852ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10232024-09-06T06:35:08.111ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7a42b808-7b1f-406b-87b7-e6aeecc73464/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3c5d89c1-0757-4549-ac27-7e3a21089176'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10242024-09-06T06:35:08.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 140.14323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
10252024-09-06T06:35:08.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 279.856726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
10262024-09-06T06:35:08.186ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_56647842-7219-46e1-9978-bdffa4d4e8cf/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='63a470b3-0451-4f99-91cb-a5fa8669d120'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10272024-09-06T06:35:08.186ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f881a8f5-60a1-4050-a3bb-fd18022091f4/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='db4839ff-d8a4-4877-ac6c-abac788dd09c'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10282024-09-06T06:35:08.207ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10292024-09-06T06:35:08.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 58.507617ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
10302024-09-06T06:35:08.217ZINFOSledAgent (ServiceManager): Booting oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
10312024-09-06T06:35:08.237ZINFOSledAgent (ServiceManager): Booting oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
10322024-09-06T06:35:08.253ZINFOSledAgent (ServiceManager): Booting oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
10332024-09-06T06:35:08.405ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
10342024-09-06T06:35:08.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 298.362615ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
10352024-09-06T06:35:08.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 297.401144ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10362024-09-06T06:35:08.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 58.696706ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
10372024-09-06T06:35:08.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 289.266267ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
10382024-09-06T06:35:08.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 310.683967ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
10392024-09-06T06:35:08.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 49.125889ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
10402024-09-06T06:35:08.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 49.032567ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
10412024-09-06T06:35:08.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 59.466033ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
10422024-09-06T06:35:08.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 42.954048ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
10432024-09-06T06:35:08.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 163.633954ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
10442024-09-06T06:35:08.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 137.264052ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
10452024-09-06T06:35:08.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 89.243155ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
10462024-09-06T06:35:08.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 52.514495ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
10472024-09-06T06:35:08.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 30.676712ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
10482024-09-06T06:35:08.794ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 95.631233ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
10492024-09-06T06:35:08.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 25.217173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
10502024-09-06T06:35:08.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 125.336704ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
10512024-09-06T06:35:08.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 227.950621ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
10522024-09-06T06:35:08.851ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 28.268068ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
10532024-09-06T06:35:08.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 269.378687ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
10542024-09-06T06:35:08.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 688.716473ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10552024-09-06T06:35:08.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 217.107029ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
10562024-09-06T06:35:08.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 396.784829ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
10572024-09-06T06:35:08.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 165.119793ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
10582024-09-06T06:35:08.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 232.468962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
10592024-09-06T06:35:08.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 73.250878ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
10602024-09-06T06:35:08.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 69.891088ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
10612024-09-06T06:35:08.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 109.947803ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
10622024-09-06T06:35:09.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 86.775249ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
10632024-09-06T06:35:09.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 265.712814ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
10642024-09-06T06:35:09.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 942.58134ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
10652024-09-06T06:35:09.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 183.412477ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
10662024-09-06T06:35:09.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 127.432132ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
10672024-09-06T06:35:09.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 395.321792ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
10682024-09-06T06:35:09.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 169.275844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
10692024-09-06T06:35:09.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 513.149744ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
10702024-09-06T06:35:09.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 304.206246ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
10712024-09-06T06:35:09.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 1.077491081s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
10722024-09-06T06:35:09.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 686.168416ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
10732024-09-06T06:35:09.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 275.883482ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
10742024-09-06T06:35:09.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 203.715664ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
10752024-09-06T06:35:09.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 212.531501ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
10762024-09-06T06:35:09.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 446.812703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
10772024-09-06T06:35:09.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 587.174486ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
10782024-09-06T06:35:09.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 448.822158ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
10792024-09-06T06:35:09.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 300.395893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
10802024-09-06T06:35:09.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.100347023s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
10812024-09-06T06:35:09.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 1.14216953s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
10822024-09-06T06:35:09.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 698.67735ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
10832024-09-06T06:35:09.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 958.301347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
10842024-09-06T06:35:09.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 514.049382ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10852024-09-06T06:35:09.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.108431512s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
10862024-09-06T06:35:09.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 1.197133568s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
10872024-09-06T06:35:09.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 1.14555244s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
10882024-09-06T06:35:09.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 923.105302ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
10892024-09-06T06:35:09.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 829.027604ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
10902024-09-06T06:35:09.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 590.017968ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
10912024-09-06T06:35:09.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 816.340264ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
10922024-09-06T06:35:10.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.43928978s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
10932024-09-06T06:35:10.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 1.467466188s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
10942024-09-06T06:35:10.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 742.711887ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
10952024-09-06T06:35:10.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 573.136298ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
10962024-09-06T06:35:10.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 1.28354413s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
10972024-09-06T06:35:10.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 893.400135ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
10982024-09-06T06:35:10.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 946.347997ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
10992024-09-06T06:35:10.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 539.7987ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
11002024-09-06T06:35:10.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.373256989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
11012024-09-06T06:35:10.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.007899169s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
11022024-09-06T06:35:10.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.253294248s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
11032024-09-06T06:35:10.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 841.537221ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
11042024-09-06T06:35:10.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 859.925458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
11052024-09-06T06:35:10.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 894.031642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
11062024-09-06T06:35:11.078ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 1.066137002s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
11072024-09-06T06:35:11.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 1.161310711s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
11082024-09-06T06:35:11.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.373620202s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
11092024-09-06T06:35:11.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 819.972642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
11102024-09-06T06:35:11.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 1.061268442s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
11112024-09-06T06:35:11.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 1.042254338s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
11122024-09-06T06:35:11.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 868.052529ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
11132024-09-06T06:35:11.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 983.733048ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
11142024-09-06T06:35:11.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 999.12137ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
11152024-09-06T06:35:11.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 793.306528ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
11162024-09-06T06:35:11.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 632.46713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
11172024-09-06T06:35:12.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.220430405s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
11182024-09-06T06:35:12.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.280945107s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
11192024-09-06T06:35:12.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 907.527034ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
11202024-09-06T06:35:12.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 877.969742ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
11212024-09-06T06:35:12.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.064884567s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
11222024-09-06T06:35:12.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 726.292276ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
11232024-09-06T06:35:12.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 916.33584ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
11242024-09-06T06:35:12.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 1.471238638s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
11252024-09-06T06:35:12.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 546.879101ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
11262024-09-06T06:35:12.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 1.079271625s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
11272024-09-06T06:35:12.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 999.814066ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
11282024-09-06T06:35:12.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 1.213080923s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
11292024-09-06T06:35:13.049ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.141610227s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
11302024-09-06T06:35:13.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 1.465428458s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
11312024-09-06T06:35:13.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.398790356s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
11322024-09-06T06:35:13.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 788.959819ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
11332024-09-06T06:35:13.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.482022948s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
11342024-09-06T06:35:13.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 1.170352292s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
11352024-09-06T06:35:13.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 1.276355567s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
11362024-09-06T06:35:13.485ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11372024-09-06T06:35:13.485ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
11382024-09-06T06:35:13.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 713.501395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
11392024-09-06T06:35:13.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 906.176092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
11402024-09-06T06:35:13.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 69.517816ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
11412024-09-06T06:35:13.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 55.581242ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
11422024-09-06T06:35:14.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 841.775297ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
11432024-09-06T06:35:14.017ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57461
11442024-09-06T06:35:14.018ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 396 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57461 req_id = 42bc15e3-8315-4580-9db4-91c721cd6940 response_code = 200 uri = /baseboard
11452024-09-06T06:35:14.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 723.628078ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
11462024-09-06T06:35:14.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 296.227669ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
11472024-09-06T06:35:14.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 628.448331ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
11482024-09-06T06:35:14.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 936.058294ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
11492024-09-06T06:35:14.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 931.276325ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
11502024-09-06T06:35:14.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 711.932125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
11512024-09-06T06:35:14.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 485.48597ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
11522024-09-06T06:35:14.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 848.712162ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
11532024-09-06T06:35:14.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 868.61111ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
11542024-09-06T06:35:14.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 937.667054ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
11552024-09-06T06:35:14.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 731.25947ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
11562024-09-06T06:35:14.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 768.58944ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
11572024-09-06T06:35:14.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 809.155916ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
11582024-09-06T06:35:14.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 1.401758042s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
11592024-09-06T06:35:14.799ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11602024-09-06T06:35:14.799ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
11612024-09-06T06:35:14.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.437973599s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
11622024-09-06T06:35:14.829ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11632024-09-06T06:35:14.830ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
11642024-09-06T06:35:14.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 900.978599ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
11652024-09-06T06:35:14.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 942.55703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
11662024-09-06T06:35:14.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.144735651s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
11672024-09-06T06:35:15.049ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 866.104233ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
11682024-09-06T06:35:15.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 34.307094ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
11692024-09-06T06:35:15.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 58.863918ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
11702024-09-06T06:35:15.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 145.026036ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
11712024-09-06T06:35:15.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 622.964763ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
11722024-09-06T06:35:15.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 61.669222ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
11732024-09-06T06:35:15.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 811.978414ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
11742024-09-06T06:35:15.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.282170975s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
11752024-09-06T06:35:15.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 152.302998ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
11762024-09-06T06:35:15.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 150.775792ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
11772024-09-06T06:35:15.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 668.981666ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
11782024-09-06T06:35:15.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 591.529353ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
11792024-09-06T06:35:15.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 575.747614ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
11802024-09-06T06:35:15.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 481.807721ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
11812024-09-06T06:35:15.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 523.728893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
11822024-09-06T06:35:15.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 1.323510231s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
11832024-09-06T06:35:15.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.312221151s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
11842024-09-06T06:35:15.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 1.463722278s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
11852024-09-06T06:35:15.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 628.710908ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
11862024-09-06T06:35:15.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 1.211280249s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
11872024-09-06T06:35:15.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 1.429558579s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
11882024-09-06T06:35:16.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 1.030919775s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
11892024-09-06T06:35:16.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 983.091204ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
11902024-09-06T06:35:16.078ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 782.282519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
11912024-09-06T06:35:16.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 1.064384524s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
11922024-09-06T06:35:16.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 737.790851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
11932024-09-06T06:35:16.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 869.017499ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
11942024-09-06T06:35:16.112ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11952024-09-06T06:35:16.112ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11962024-09-06T06:35:16.112ZINFOSledAgent (ServiceManager): Booting oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
11972024-09-06T06:35:16.118ZINFOSledAgent (ServiceManager): Booting oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
11982024-09-06T06:35:16.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 633.4845ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
11992024-09-06T06:35:16.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 1.398943926s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
12002024-09-06T06:35:16.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.134659742s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
12012024-09-06T06:35:16.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 55.675926ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12022024-09-06T06:35:16.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 56.173528ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12032024-09-06T06:35:16.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 124.200234ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12042024-09-06T06:35:16.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 82.525057ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12052024-09-06T06:35:16.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 922.593205ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
12062024-09-06T06:35:16.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 283.005542ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12072024-09-06T06:35:16.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 209.580197ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12082024-09-06T06:35:16.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.430318095s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
12092024-09-06T06:35:16.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 1.476625137s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
12102024-09-06T06:35:16.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 230.792232ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12112024-09-06T06:35:16.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 334.964054ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12122024-09-06T06:35:16.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 1.233269562s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
12132024-09-06T06:35:17.049ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.197359696s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
12142024-09-06T06:35:17.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 1.42188561s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
12152024-09-06T06:35:17.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 616.193685ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
12162024-09-06T06:35:17.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.040964071s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
12172024-09-06T06:35:17.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 1.047502426s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
12182024-09-06T06:35:17.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.141759096s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12192024-09-06T06:35:17.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 604.364473ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
12202024-09-06T06:35:17.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 1.377517711s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
12212024-09-06T06:35:17.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 1.292914348s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
12222024-09-06T06:35:17.215ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12232024-09-06T06:35:17.215ZINFOSledAgent (ServiceManager): Booting oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12242024-09-06T06:35:17.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 693.46566ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12252024-09-06T06:35:17.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 648.528344ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
12262024-09-06T06:35:17.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 1.397712344s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
12272024-09-06T06:35:17.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.063215716s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
12282024-09-06T06:35:17.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 45.121624ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12292024-09-06T06:35:17.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 836.713686ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
12302024-09-06T06:35:17.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 96.976282ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12312024-09-06T06:35:17.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 233.683059ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12322024-09-06T06:35:17.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 1.080692484s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
12332024-09-06T06:35:17.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.141379385s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
12342024-09-06T06:35:17.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 610.01603ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
12352024-09-06T06:35:17.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 407.818775ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12362024-09-06T06:35:17.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 661.62777ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
12372024-09-06T06:35:17.956ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 660.637254ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12382024-09-06T06:35:18.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.24622671s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
12392024-09-06T06:35:18.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.348115025s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
12402024-09-06T06:35:18.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 1.141739197s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
12412024-09-06T06:35:18.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 559.462681ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
12422024-09-06T06:35:18.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.168525569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
12432024-09-06T06:35:18.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.160424743s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12442024-09-06T06:35:18.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 836.977269ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12452024-09-06T06:35:18.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 1.150709559s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
12462024-09-06T06:35:18.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 977.070071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
12472024-09-06T06:35:18.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 1.152519749s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
12482024-09-06T06:35:18.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 1.264241904s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
12492024-09-06T06:35:18.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 991.132739ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
12502024-09-06T06:35:18.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.481295145s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
12512024-09-06T06:35:18.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 942.708916ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
12522024-09-06T06:35:18.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 689.520031ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
12532024-09-06T06:35:18.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 1.486410802s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12542024-09-06T06:35:18.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 1.120989155s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
12552024-09-06T06:35:18.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 602.20347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
12562024-09-06T06:35:18.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 695.500045ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
12572024-09-06T06:35:18.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.280645745s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
12582024-09-06T06:35:19.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 1.122793842s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12592024-09-06T06:35:19.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 876.584851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
12602024-09-06T06:35:19.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 1.03835234s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
12612024-09-06T06:35:19.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 1.417452804s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
12622024-09-06T06:35:19.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 997.026706ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
12632024-09-06T06:35:19.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 1.379042311s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
12642024-09-06T06:35:19.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.211364742s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
12652024-09-06T06:35:19.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 514.434409ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12662024-09-06T06:35:19.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 966.990728ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
12672024-09-06T06:35:19.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 1.088690199s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
12682024-09-06T06:35:19.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 976.796238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
12692024-09-06T06:35:19.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 1.430179827s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
12702024-09-06T06:35:19.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 547.008703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
12712024-09-06T06:35:19.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 546.725545ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
12722024-09-06T06:35:19.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 613.763883ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
12732024-09-06T06:35:19.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 742.587151ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
12742024-09-06T06:35:19.912ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.210158596s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
12752024-09-06T06:35:19.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 502.613103ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12762024-09-06T06:35:19.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.366121211s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
12772024-09-06T06:35:20.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.439485999s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
12782024-09-06T06:35:20.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 1.113840506s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
12792024-09-06T06:35:20.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 1.495664418s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
12802024-09-06T06:35:20.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.200379494s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
12812024-09-06T06:35:20.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 617.85355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
12822024-09-06T06:35:20.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 1.302424794s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
12832024-09-06T06:35:20.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.443967815s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
12842024-09-06T06:35:20.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 773.765942ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
12852024-09-06T06:35:20.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 772.62953ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
12862024-09-06T06:35:20.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 709.572672ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
12872024-09-06T06:35:20.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 1.18148422s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
12882024-09-06T06:35:20.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.488423061s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
12892024-09-06T06:35:20.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 928.723863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
12902024-09-06T06:35:20.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 506.747922ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
12912024-09-06T06:35:20.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.082523543s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
12922024-09-06T06:35:20.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 916.623713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
12932024-09-06T06:35:20.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 528.621433ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
12942024-09-06T06:35:20.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 1.458596846s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
12952024-09-06T06:35:20.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 562.520006ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
12962024-09-06T06:35:21.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 727.039506ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
12972024-09-06T06:35:21.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 716.037117ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
12982024-09-06T06:35:21.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 731.905849ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
12992024-09-06T06:35:21.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 1.010066981s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
13002024-09-06T06:35:21.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 1.362559371s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13012024-09-06T06:35:21.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 1.418549914s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
13022024-09-06T06:35:21.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 1.235465464s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
13032024-09-06T06:35:21.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 977.452296ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
13042024-09-06T06:35:21.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 1.196333276s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
13052024-09-06T06:35:21.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 574.126264ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
13062024-09-06T06:35:21.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.058675929s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
13072024-09-06T06:35:21.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 856.136853ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13082024-09-06T06:35:21.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 610.864594ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13092024-09-06T06:35:21.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 1.493423254s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
13102024-09-06T06:35:21.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 529.088879ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
13112024-09-06T06:35:21.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 666.92356ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
13122024-09-06T06:35:21.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.0859474s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
13132024-09-06T06:35:21.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 1.081332601s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
13142024-09-06T06:35:21.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 876.239463ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
13152024-09-06T06:35:21.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 607.229521ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
13162024-09-06T06:35:21.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.286167569s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
13172024-09-06T06:35:22.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 654.942981ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
13182024-09-06T06:35:22.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 1.240413849s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
13192024-09-06T06:35:22.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 1.078659825s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
13202024-09-06T06:35:22.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 771.19773ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13212024-09-06T06:35:22.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 799.365125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
13222024-09-06T06:35:22.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31") failed: Property not found. retry in 1.130647956s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
13232024-09-06T06:35:22.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 591.147407ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
13242024-09-06T06:35:22.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 824.636293ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13252024-09-06T06:35:22.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 905.241185ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
13262024-09-06T06:35:22.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 545.829405ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13272024-09-06T06:35:22.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 707.766296ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
13282024-09-06T06:35:22.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 711.852021ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
13292024-09-06T06:35:22.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 1.309668725s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
13302024-09-06T06:35:22.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569") failed: Property not found. retry in 1.322696025s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
13312024-09-06T06:35:22.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 1.389945783s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
13322024-09-06T06:35:22.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc") failed: Property not found. retry in 1.226641905s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
13332024-09-06T06:35:22.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8") failed: Property not found. retry in 1.325538119s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
13342024-09-06T06:35:22.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66") failed: Property not found. retry in 840.970689ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
13352024-09-06T06:35:23.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a") failed: Property not found. retry in 1.49655291s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
13362024-09-06T06:35:23.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 511.426616ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13372024-09-06T06:35:23.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 693.94631ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
13382024-09-06T06:35:23.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 685.286735ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
13392024-09-06T06:35:23.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 517.965312ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13402024-09-06T06:35:23.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 1.335906632s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13412024-09-06T06:35:23.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.19336588s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
13422024-09-06T06:35:23.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181") failed: Property not found. retry in 769.538384ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
13432024-09-06T06:35:23.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950") failed: Property not found. retry in 849.287931ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
13442024-09-06T06:35:23.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7") failed: Property not found. retry in 1.295736539s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
13452024-09-06T06:35:23.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 1.026843774s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
13462024-09-06T06:35:23.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421") failed: Property not found. retry in 835.371663ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
13472024-09-06T06:35:23.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 1.228744856s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13482024-09-06T06:35:23.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 1.018901347s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13492024-09-06T06:35:23.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598") failed: Property not found. retry in 1.165441523s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
13502024-09-06T06:35:23.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24") failed: Property not found. retry in 1.256433414s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
13512024-09-06T06:35:24.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5") failed: Property not found. retry in 897.818775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
13522024-09-06T06:35:24.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 1.048726656s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
13532024-09-06T06:35:24.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 859.098198ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
13542024-09-06T06:35:24.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 578.541617ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
13552024-09-06T06:35:24.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 826.92613ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13562024-09-06T06:35:24.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 845.956788ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13572024-09-06T06:35:24.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 1.002659596s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13582024-09-06T06:35:25.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.141002246s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
13592024-09-06T06:35:25.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 1.441469548s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
13602024-09-06T06:35:25.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 1.497556162s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
13612024-09-06T06:35:25.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 648.062816ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13622024-09-06T06:35:25.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 800.692064ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13632024-09-06T06:35:25.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 726.04201ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13642024-09-06T06:35:26.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 1.46376235s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13652024-09-06T06:35:26.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.050262494s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
13662024-09-06T06:35:26.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 673.905165ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13672024-09-06T06:35:26.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 1.028105993s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13682024-09-06T06:35:26.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 1.27702808s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
13692024-09-06T06:35:26.901ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d") failed: Property not found. retry in 1.481702201s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
13702024-09-06T06:35:27.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45") failed: Property not found. retry in 1.461015577s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13712024-09-06T06:35:27.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476") failed: Property not found. retry in 1.099163129s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
13722024-09-06T06:35:27.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 503.460422ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13732024-09-06T06:35:27.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189") failed: Property not found. retry in 1.142194907s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13742024-09-06T06:35:27.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618") failed: Property not found. retry in 1.479422384s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
13752024-09-06T06:35:28.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c") failed: Property not found. retry in 937.198826ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13762024-09-06T06:35:28.705ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.898540034s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
13772024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_nexus_f4c090fd-0141-407e-a547-8163f49da46c
13782024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_clickhouse_c99a3832-620a-48f7-b103-35c8b28bf618
13792024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_pantry_e9284324-b820-4122-bd9c-c86c1a78e569
13802024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_nexus_e9c280ee-8313-41ab-8fa4-df346c954189
13812024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_7ea4e3c3-4afc-46a9-9d22-0f924a266ab8
13822024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_external_dns_bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d
13832024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_oximeter_e919b437-ca84-4c31-970f-ae903b0601dc
13842024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_dcc2c2b5-9bd3-4b61-9a34-b6f0b0baee24
13852024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_789dddab-4b92-4605-841e-47bdff42904a
13862024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_e90b2a1c-bdff-4e4e-b069-89ce02023950
13872024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_pantry_6395a09e-9850-4d3d-bced-7fead6fd0c31
13882024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_6a0f50b7-0301-4b31-ab4b-e47d8d5e2421
13892024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_1d333596-f29b-425b-a7df-0e33acb1d598
13902024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_89028fef-e2fe-45a6-a453-d7e529ee2181
13912024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_e11f7659-c40d-4b11-8887-429b1186d7b7
13922024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_ce5f16f3-697d-4f00-ade9-701fa0baf1a5
13932024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_nexus_5d85f2d3-87bb-4fce-a9bf-1f1805221476
13942024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_pantry_682aec46-4e71-4cbe-a6b3-ae4527771b66
13952024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_external_dns_81c3dd8c-a34f-489a-91c0-599e726f0b45
13962024-09-06T06:35:29.505ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
13972024-09-06T06:35:29.510ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
13982024-09-06T06:35:29.514ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25985008 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:56930 req_id = 03f404cb-c654-42f1-9b6d-0e10c3eff23a response_code = 204 uri = /omicron-zones
13992024-09-06T06:35:29.518ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1322
14002024-09-06T06:35:29.518ZINFOSledAgent (RSS): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
14012024-09-06T06:35:29.518ZINFOSledAgent (RSS): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
14022024-09-06T06:35:29.520ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
14032024-09-06T06:35:29.520ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:720
14042024-09-06T06:35:29.556ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:887
14052024-09-06T06:35:29.557ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14062024-09-06T06:35:29.653ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14072024-09-06T06:35:29.793ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14082024-09-06T06:35:30.009ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14092024-09-06T06:35:30.134ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14102024-09-06T06:35:30.275ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14112024-09-06T06:35:30.576ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14122024-09-06T06:35:30.849ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14132024-09-06T06:35:31.350ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14142024-09-06T06:35:31.991ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14152024-09-06T06:35:32.604ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14162024-09-06T06:35:33.398ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14172024-09-06T06:35:34.381ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14182024-09-06T06:35:35.573ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14192024-09-06T06:35:36.372ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14202024-09-06T06:35:37.359ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14212024-09-06T06:35:38.639ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 13.478066332s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
14222024-09-06T06:35:38.882ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14232024-09-06T06:35:41.271ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14242024-09-06T06:35:42.516ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14252024-09-06T06:35:44.053ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43030
14262024-09-06T06:35:44.053ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 36 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43030 req_id = 4322d778-218b-439b-9abd-36b73afa84b7 response_code = 200 uri = /baseboard
14272024-09-06T06:35:45.583ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/99cd7c46-10a2-48c7-8118-fbe80ea47643/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
14282024-09-06T06:35:50.373ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "203a4fa9-baee-4bfe-9c9c-e3887df8b501", "content-length": "173", "date": "Fri, 06 Sep 2024 06:35:50 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"99cd7c46-10a2-48c7-8118-fbe80ea47643\"", request_id: "203a4fa9-baee-4bfe-9c9c-e3887df8b501" } file = sled-agent/src/rack_setup/service.rs:949
14292024-09-06T06:35:56.719ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "b1fb94a8-ade3-4a38-8c4b-a33513478f3b", "content-length": "173", "date": "Fri, 06 Sep 2024 06:35:56 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"58853ca1-8d5a-4912-87e6-57de713a602c\"", request_id: "b1fb94a8-ade3-4a38-8c4b-a33513478f3b" } file = sled-agent/src/rack_setup/service.rs:949
14302024-09-06T06:36:03.185ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "9b4fec11-97c2-4061-83f2-ee4bb4628bdb", "content-length": "173", "date": "Fri, 06 Sep 2024 06:36:03 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"58853ca1-8d5a-4912-87e6-57de713a602c\"", request_id: "9b4fec11-97c2-4061-83f2-ee4bb4628bdb" } file = sled-agent/src/rack_setup/service.rs:949
14312024-09-06T06:36:04.549ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
14322024-09-06T06:36:04.675ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58779
14332024-09-06T06:36:04.676ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14342024-09-06T06:36:04.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 204644 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58779 req_id = febf4754-220d-4db7-905f-6b56ce6518c0 response_code = 200 uri = /inventory
14352024-09-06T06:36:04.889ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14362024-09-06T06:36:04.889ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-zones.json file = common/src/ledger.rs:177
14372024-09-06T06:36:04.889ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-zones.json file = common/src/ledger.rs:177
14382024-09-06T06:36:04.890ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 881 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58779 req_id = 06991489-5fff-4312-a22a-91a641ec90cd response_code = 200 uri = /omicron-zones
14392024-09-06T06:36:11.037ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39810
14402024-09-06T06:36:11.038ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14412024-09-06T06:36:11.038ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14422024-09-06T06:36:11.038ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14432024-09-06T06:36:11.038ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14442024-09-06T06:36:11.038ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
14452024-09-06T06:36:11.038ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14462024-09-06T06:36:11.038ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14472024-09-06T06:36:11.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 418 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39810 req_id = 2ba0554a-647b-4702-8d07-2df60a732e70 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14482024-09-06T06:36:11.039ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:959
14492024-09-06T06:36:11.219ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49505
14502024-09-06T06:36:11.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14512024-09-06T06:36:11.446ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 226873 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49505 req_id = adbaf6e2-580f-42d5-ad5e-9ffade89f9f0 response_code = 200 uri = /inventory
14522024-09-06T06:36:11.446ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14532024-09-06T06:36:11.446ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/config/omicron-zones.json file = common/src/ledger.rs:177
14542024-09-06T06:36:11.447ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/config/omicron-zones.json file = common/src/ledger.rs:177
14552024-09-06T06:36:11.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 835 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49505 req_id = 03cf1474-8b5e-4433-972a-e9c936c3cfb3 response_code = 200 uri = /omicron-zones
14562024-09-06T06:36:11.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44221
14572024-09-06T06:36:11.488ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14582024-09-06T06:36:11.488ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14592024-09-06T06:36:11.488ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14602024-09-06T06:36:11.488ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14612024-09-06T06:36:11.488ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
14622024-09-06T06:36:11.488ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14632024-09-06T06:36:11.488ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14642024-09-06T06:36:11.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1567 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44221 req_id = 1d33e0d8-404e-4f09-9d8f-48fd1fcba122 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14652024-09-06T06:36:11.540ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42153
14662024-09-06T06:36:11.540ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14672024-09-06T06:36:11.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14682024-09-06T06:36:11.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14692024-09-06T06:36:11.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14702024-09-06T06:36:11.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
14712024-09-06T06:36:11.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14722024-09-06T06:36:11.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14732024-09-06T06:36:11.541ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 291 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42153 req_id = a07d1ff7-339b-4a25-8d37-77305a011f75 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14742024-09-06T06:36:11.671ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34253
14752024-09-06T06:36:11.672ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14762024-09-06T06:36:11.672ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14772024-09-06T06:36:11.672ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14782024-09-06T06:36:11.672ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14792024-09-06T06:36:11.672ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
14802024-09-06T06:36:11.672ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14812024-09-06T06:36:11.672ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14822024-09-06T06:36:11.672ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 931 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34253 req_id = 83a78752-b98b-4c81-ab3a-8037c9b81784 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14832024-09-06T06:36:14.089ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44757
14842024-09-06T06:36:14.089ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 687 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44757 req_id = 5d271690-9e96-4073-9086-b5155d5aec99 response_code = 200 uri = /baseboard
14852024-09-06T06:36:21.443ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44325
14862024-09-06T06:36:21.444ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 680 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44325 req_id = d1163431-3d6b-45b3-a700-bdccee15126a response_code = 200 uri = /vpc-routes
14872024-09-06T06:36:21.544ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55177
14882024-09-06T06:36:21.544ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55177 req_id = 58e2ae50-c851-4547-9d5e-781122499495 response_code = 200 uri = /v2p
14892024-09-06T06:36:21.589ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 714 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44325 req_id = d597c67f-ba29-45c5-aaf5-d86533b62645 response_code = 204 uri = /vpc-routes
14902024-09-06T06:36:21.825ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61660
14912024-09-06T06:36:21.825ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
14922024-09-06T06:36:21.901ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
14932024-09-06T06:36:21.954ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129120 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61660 req_id = e69eaf95-4427-433e-a899-98b2081728ea response_code = 204 uri = /switch-ports
14942024-09-06T06:36:21.954ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61660 req_id = 31ccd780-b01d-4028-ab45-5d6b6ae0a24e response_code = 200 uri = /network-bootstore-config
14952024-09-06T06:36:22.141ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
14962024-09-06T06:36:22.141ZINFOSledAgent (bootstore): Writing ledger to /pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
14972024-09-06T06:36:22.147ZINFOSledAgent (bootstore): Writing ledger to /pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
14982024-09-06T06:36:22.160ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
14992024-09-06T06:36:22.160ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 19566 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61660 req_id = edf25be0-449e-4606-99f9-0ebb49db5245 response_code = 204 uri = /network-bootstore-config
15002024-09-06T06:36:22.172ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38191
15012024-09-06T06:36:22.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 393 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38191 req_id = 05f1a160-eaeb-4d25-9f9d-c017e8c6ffe4 response_code = 200 uri = /vpc-routes
15022024-09-06T06:36:22.213ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:38191 req_id = 1c1dec8d-dd98-4b5d-9286-456d306f76e4 response_code = 204 uri = /vpc-routes
15032024-09-06T06:36:22.288ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53767
15042024-09-06T06:36:22.288ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53767 req_id = f41d16b3-c897-47bd-8c24-0be696cefc35 response_code = 200 uri = /vpc-routes
15052024-09-06T06:36:22.289ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45778
15062024-09-06T06:36:22.290ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15072024-09-06T06:36:22.338ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15082024-09-06T06:36:22.398ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109231 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45778 req_id = bc1201b9-edec-4a6d-a258-0a5842d56dce response_code = 204 uri = /switch-ports
15092024-09-06T06:36:22.398ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:38689 remote_addr = [fd00:1122:3344:101::d]:56516 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c
15102024-09-06T06:36:22.398ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42700
15112024-09-06T06:36:22.398ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54948
15122024-09-06T06:36:22.399ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 240 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42700 req_id = b529adab-b425-45a3-bb79-f017ad45c193 response_code = 200 uri = /v2p
15132024-09-06T06:36:22.399ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 308 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53767 req_id = 185d61de-32a1-41b4-8525-1af00a47d0be response_code = 204 uri = /vpc-routes
15142024-09-06T06:36:22.399ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 225 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54948 req_id = e489ed67-8b53-4b63-885e-13fd584dba22 response_code = 200 uri = /v2p
15152024-09-06T06:36:22.400ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 216 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45778 req_id = 30fd2ebc-a1e6-42b5-b15d-861ed52319e2 response_code = 200 uri = /network-bootstore-config
15162024-09-06T06:36:22.414ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 15606 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 4fe04ea8-892d-462f-80b4-b4dc9c7f7653 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
15172024-09-06T06:36:22.544ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49563
15182024-09-06T06:36:22.544ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15192024-09-06T06:36:22.581ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15202024-09-06T06:36:22.635ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90839 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49563 req_id = 7bc6e5e8-55cf-4060-9cbf-39d4ec16a698 response_code = 204 uri = /switch-ports
15212024-09-06T06:36:22.638ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 864 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49563 req_id = fb34fd0a-4ded-40df-a14f-7f6ff94ea4ee response_code = 200 uri = /network-bootstore-config
15222024-09-06T06:36:25.853ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44812
15232024-09-06T06:36:25.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44812 req_id = 60d712f2-c453-43e0-8f9d-42c46bbb266b response_code = 200 uri = /vpc-routes
15242024-09-06T06:36:25.863ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44812 req_id = 52629869-65e9-48e2-9a1e-b4a332897261 response_code = 204 uri = /vpc-routes
15252024-09-06T06:36:44.131ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44764
15262024-09-06T06:36:44.131ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 563 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44764 req_id = c8b7f16d-21a5-4f82-8734-87a7f4d13930 response_code = 200 uri = /baseboard
15272024-09-06T06:36:51.476ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40398
15282024-09-06T06:36:51.476ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40398 req_id = c3814d66-a15a-4a56-b15f-8ca7bc6de646 response_code = 200 uri = /vpc-routes
15292024-09-06T06:36:51.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40398 req_id = c8ce7c6b-8fb2-4528-a3b4-960e313dbe57 response_code = 204 uri = /vpc-routes
15302024-09-06T06:36:51.660ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41322
15312024-09-06T06:36:51.661ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41322 req_id = a16bf782-aedb-45d2-9bf2-f3b1735482e7 response_code = 200 uri = /v2p
15322024-09-06T06:36:51.922ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63347
15332024-09-06T06:36:51.923ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15342024-09-06T06:36:51.960ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15352024-09-06T06:36:52.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115546 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63347 req_id = 7f752fc4-1193-49db-9977-3499d1ae91f8 response_code = 204 uri = /switch-ports
15362024-09-06T06:36:52.040ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1505 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63347 req_id = a5166f3c-d5bb-4296-aeef-02c14734f012 response_code = 200 uri = /network-bootstore-config
15372024-09-06T06:36:52.233ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54904
15382024-09-06T06:36:52.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 906 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54904 req_id = e8d09c1a-8365-4e32-9a48-ec88f1da4768 response_code = 200 uri = /vpc-routes
15392024-09-06T06:36:52.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54904 req_id = 15a14d2e-7b5c-48ad-875f-3af3a2e78b04 response_code = 204 uri = /vpc-routes
15402024-09-06T06:36:52.337ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6990 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 34203cc3-8ffd-443a-9ce4-46b44f337f48 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
15412024-09-06T06:36:52.343ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41235
15422024-09-06T06:36:52.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1000 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41235 req_id = e7e89f1c-f6eb-4d95-8df4-af8438ddd989 response_code = 200 uri = /vpc-routes
15432024-09-06T06:36:52.354ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39252
15442024-09-06T06:36:52.355ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 997 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39252 req_id = 22398dcc-c231-4ab2-b37b-1be96ee0035e response_code = 200 uri = /v2p
15452024-09-06T06:36:52.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41235 req_id = 9e123248-ed18-4a5d-afa6-efaf9f3c1f77 response_code = 204 uri = /vpc-routes
15462024-09-06T06:36:52.466ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56334
15472024-09-06T06:36:52.466ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15482024-09-06T06:36:52.504ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15492024-09-06T06:36:52.563ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96912 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56334 req_id = 79b87b6c-0f9b-480a-842b-29954e29ef24 response_code = 204 uri = /switch-ports
15502024-09-06T06:36:52.563ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56334 req_id = 83df9030-c012-443f-b845-2f4a10be8e4a response_code = 200 uri = /network-bootstore-config
15512024-09-06T06:36:52.626ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61176
15522024-09-06T06:36:52.626ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61176 req_id = 0e6a4e14-e7a9-495a-b2e7-0395c4eef3fb response_code = 200 uri = /v2p
15532024-09-06T06:36:52.782ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62557
15542024-09-06T06:36:52.783ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15552024-09-06T06:36:52.821ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15562024-09-06T06:36:52.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93506 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:62557 req_id = e89f07c0-a1bd-4eb0-945e-12ef0825ca1a response_code = 204 uri = /switch-ports
15572024-09-06T06:36:52.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62557 req_id = d9c125f9-b62c-4887-b7dc-a0bda80b4b60 response_code = 200 uri = /network-bootstore-config
15582024-09-06T06:37:14.164ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51070
15592024-09-06T06:37:14.164ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51070 req_id = f3b24b85-e3ba-472e-af99-4eff8e2b3166 response_code = 200 uri = /baseboard
15602024-09-06T06:37:21.347ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40263
15612024-09-06T06:37:21.348ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40263 req_id = f52cf3aa-5a4d-48e7-992a-090cdd6743fe response_code = 200 uri = /vpc-routes
15622024-09-06T06:37:21.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40263 req_id = f16bf130-3d89-423e-827a-b4c77230f3da response_code = 204 uri = /vpc-routes
15632024-09-06T06:37:21.396ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60735
15642024-09-06T06:37:21.397ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15652024-09-06T06:37:21.435ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15662024-09-06T06:37:21.492ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95518 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60735 req_id = 8cd071b7-2e17-4b8e-b2ef-18618346e274 response_code = 204 uri = /switch-ports
15672024-09-06T06:37:21.492ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60735 req_id = 79316ef3-c4e8-4adc-b973-73b34156b4c0 response_code = 200 uri = /network-bootstore-config
15682024-09-06T06:37:21.526ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37584
15692024-09-06T06:37:21.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 648 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37584 req_id = 7d33f6e0-ed2b-4603-b396-a80b350615c7 response_code = 200 uri = /v2p
15702024-09-06T06:37:22.104ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34883
15712024-09-06T06:37:22.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1673 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34883 req_id = 6858d80f-054a-42dd-867b-02c26db83d7f response_code = 200 uri = /vpc-routes
15722024-09-06T06:37:22.122ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1487 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34883 req_id = 49122b72-8cf1-4992-993c-c9e605ca0126 response_code = 204 uri = /vpc-routes
15732024-09-06T06:37:22.213ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:32773
15742024-09-06T06:37:22.213ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:32773 req_id = d34b5aa4-0399-484a-a4d3-0c2c117520e2 response_code = 200 uri = /v2p
15752024-09-06T06:37:22.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52373
15762024-09-06T06:37:22.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52373 req_id = 9a87fc31-1bd2-41dc-b03d-3b0c8b12522a response_code = 200 uri = /vpc-routes
15772024-09-06T06:37:22.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 669 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52373 req_id = de8f8c20-db57-444f-b9ba-ea33bee2adc2 response_code = 204 uri = /vpc-routes
15782024-09-06T06:37:22.338ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7932 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 6c836e4d-a27d-48c2-942f-a105c71f3e89 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
15792024-09-06T06:37:22.386ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54445
15802024-09-06T06:37:22.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54445 req_id = 71c67e73-faf5-4ec0-9fa9-9d612023958f response_code = 200 uri = /v2p
15812024-09-06T06:37:22.392ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45527
15822024-09-06T06:37:22.392ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15832024-09-06T06:37:22.431ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15842024-09-06T06:37:22.484ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92219 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45527 req_id = 1896401e-2016-4b73-8807-e12d13d18a5a response_code = 204 uri = /switch-ports
15852024-09-06T06:37:22.484ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45631
15862024-09-06T06:37:22.484ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
15872024-09-06T06:37:22.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3602 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45527 req_id = 4e7516a2-78dd-4cc1-8a06-25b479387ead response_code = 200 uri = /network-bootstore-config
15882024-09-06T06:37:22.519ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
15892024-09-06T06:37:22.573ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88844 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45631 req_id = 0f34e66a-7549-4484-a2d2-a419ed17cd17 response_code = 204 uri = /switch-ports
15902024-09-06T06:37:22.573ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45631 req_id = d05ccf62-8b4f-4d98-9a85-141171a8be33 response_code = 200 uri = /network-bootstore-config
15912024-09-06T06:37:44.198ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58856
15922024-09-06T06:37:44.199ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58856 req_id = 3f78df5a-d656-44d8-b663-486a7ab9ea02 response_code = 200 uri = /baseboard
15932024-09-06T06:37:51.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37955
15942024-09-06T06:37:51.522ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 744 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37955 req_id = 4bf6bf80-d490-41d2-af7d-cd3dbbdd4f3c response_code = 200 uri = /vpc-routes
15952024-09-06T06:37:51.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37955 req_id = 6fbd6e6e-9298-4b4d-9518-f4a95c538b4f response_code = 204 uri = /vpc-routes
15962024-09-06T06:37:51.600ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35044
15972024-09-06T06:37:51.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35044 req_id = f88cf0c3-d95a-4cb5-b0eb-a86e8bc76bf6 response_code = 200 uri = /v2p
15982024-09-06T06:37:51.660ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45435
15992024-09-06T06:37:51.660ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16002024-09-06T06:37:51.698ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16012024-09-06T06:37:51.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97533 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:45435 req_id = b7c33de6-8f9f-42ed-8da2-b765c94028fa response_code = 204 uri = /switch-ports
16022024-09-06T06:37:51.759ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 726 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45435 req_id = 55ba1700-4d39-4469-9eb6-d4c8ccf0d6f0 response_code = 200 uri = /network-bootstore-config
16032024-09-06T06:37:52.197ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50095
16042024-09-06T06:37:52.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50095 req_id = c8f23c2d-8042-47d6-9cf2-ae67d3708bd7 response_code = 200 uri = /v2p
16052024-09-06T06:37:52.337ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6553 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 3d3cd753-4d50-4212-b8fd-e9819e6f99ff response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
16062024-09-06T06:37:52.380ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44634
16072024-09-06T06:37:52.380ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44634 req_id = 4f3a2a4b-b4c2-4ae8-9e6a-627aa72b6241 response_code = 200 uri = /vpc-routes
16082024-09-06T06:37:52.394ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50666
16092024-09-06T06:37:52.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50666 req_id = 476c68c5-db52-4a2c-8ba0-e30bdc7838f1 response_code = 200 uri = /vpc-routes
16102024-09-06T06:37:52.403ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50666 req_id = b650f9e7-4c3e-4a9c-997e-ab39de398948 response_code = 204 uri = /vpc-routes
16112024-09-06T06:37:52.481ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47901
16122024-09-06T06:37:52.482ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16132024-09-06T06:37:52.522ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16142024-09-06T06:37:52.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99448 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47901 req_id = c8547707-3325-4d6a-bfaa-0b4b3071fff2 response_code = 204 uri = /switch-ports
16152024-09-06T06:37:52.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47901 req_id = 37dd7be6-5b56-47b6-8db6-b7e9dd08ef23 response_code = 200 uri = /network-bootstore-config
16162024-09-06T06:37:52.594ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59856
16172024-09-06T06:37:52.595ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1015 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59856 req_id = 78229999-b4d9-4706-a86d-914a215a90a6 response_code = 200 uri = /v2p
16182024-09-06T06:37:52.601ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44634 req_id = 6a02ab86-9220-4697-8e8a-b2954ba1192e response_code = 204 uri = /vpc-routes
16192024-09-06T06:37:52.720ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63274
16202024-09-06T06:37:52.720ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16212024-09-06T06:37:52.758ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16222024-09-06T06:37:52.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91199 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63274 req_id = 92fe546d-0d84-4f97-a65d-260275be72ac response_code = 204 uri = /switch-ports
16232024-09-06T06:37:52.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63274 req_id = 5fa76144-5761-4ab0-8e57-80b193612037 response_code = 200 uri = /network-bootstore-config
16242024-09-06T06:38:14.236ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65083
16252024-09-06T06:38:14.236ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65083 req_id = a87f7876-171e-4eeb-bb3f-63a0d81b46f1 response_code = 200 uri = /baseboard
16262024-09-06T06:38:21.349ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33591
16272024-09-06T06:38:21.349ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 250 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33591 req_id = 7aa71e89-3758-4e9b-bba1-363eb1d5a179 response_code = 200 uri = /v2p
16282024-09-06T06:38:21.391ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59635
16292024-09-06T06:38:21.391ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59635 req_id = 1a96681d-ffeb-4c30-8e1a-171a134d49c0 response_code = 200 uri = /vpc-routes
16302024-09-06T06:38:21.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59635 req_id = dd5a602c-c22f-4b58-a567-3d9bf16a78e8 response_code = 204 uri = /vpc-routes
16312024-09-06T06:38:21.549ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36789
16322024-09-06T06:38:21.550ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16332024-09-06T06:38:21.587ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16342024-09-06T06:38:21.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91786 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36789 req_id = e9267259-9b95-4adc-9a8b-835c15211aa2 response_code = 204 uri = /switch-ports
16352024-09-06T06:38:21.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36789 req_id = f61f9b35-fbe4-41c9-b81e-8ce22833890d response_code = 200 uri = /network-bootstore-config
16362024-09-06T06:38:22.155ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56973
16372024-09-06T06:38:22.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56973 req_id = 5189cb91-1c11-44c0-bab8-99c11ced2e95 response_code = 200 uri = /vpc-routes
16382024-09-06T06:38:22.163ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56973 req_id = 103705fe-3fb6-4d65-90cc-16eae4fef7fa response_code = 204 uri = /vpc-routes
16392024-09-06T06:38:22.253ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51150
16402024-09-06T06:38:22.254ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 838 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51150 req_id = 13c7ab3c-e2ae-4827-ab6a-26c21673a7c8 response_code = 200 uri = /v2p
16412024-09-06T06:38:22.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56625
16422024-09-06T06:38:22.257ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16432024-09-06T06:38:22.294ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16442024-09-06T06:38:22.354ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97175 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56625 req_id = 648ec76c-fe99-420b-a373-92c9e03395e2 response_code = 204 uri = /switch-ports
16452024-09-06T06:38:22.354ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44037
16462024-09-06T06:38:22.355ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 254 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44037 req_id = cc585396-e7b8-404d-b5cb-149a92e67a06 response_code = 200 uri = /vpc-routes
16472024-09-06T06:38:22.355ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 306 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56625 req_id = 77697b52-b7d8-4305-8d69-8d28c878baad response_code = 200 uri = /network-bootstore-config
16482024-09-06T06:38:22.364ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9879 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 8620d695-6f97-4b4a-a45b-f0b2089245c5 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
16492024-09-06T06:38:22.423ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47550
16502024-09-06T06:38:22.423ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47550 req_id = c4433e72-830f-4106-a7f7-5b417f202b0c response_code = 200 uri = /v2p
16512024-09-06T06:38:22.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44037 req_id = 5eedbd16-8e62-463f-9b76-f0e8ef4bbfa4 response_code = 204 uri = /vpc-routes
16522024-09-06T06:38:22.468ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54093
16532024-09-06T06:38:22.468ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16542024-09-06T06:38:22.507ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16552024-09-06T06:38:22.565ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97211 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54093 req_id = 97128859-39d3-4004-a23c-55f99aab1cde response_code = 204 uri = /switch-ports
16562024-09-06T06:38:22.566ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 754 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54093 req_id = bb5f942f-13a1-4952-80e0-5396c4e5d959 response_code = 200 uri = /network-bootstore-config
16572024-09-06T06:38:38.353ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
16582024-09-06T06:38:44.269ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54278
16592024-09-06T06:38:44.269ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54278 req_id = d56c70c8-b787-4539-8654-deb43ee6c49e response_code = 200 uri = /baseboard
16602024-09-06T06:38:51.576ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57036
16612024-09-06T06:38:51.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57036 req_id = 2d457e55-c884-43ea-973f-008d248bd29a response_code = 200 uri = /vpc-routes
16622024-09-06T06:38:51.648ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48324
16632024-09-06T06:38:51.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48324 req_id = 131a47f9-8702-44c9-b3d8-c4c64ac72c10 response_code = 200 uri = /v2p
16642024-09-06T06:38:51.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57036 req_id = 8bb18bde-952d-4961-be1f-719cda6ae8bd response_code = 204 uri = /vpc-routes
16652024-09-06T06:38:51.853ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62219
16662024-09-06T06:38:51.853ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16672024-09-06T06:38:51.891ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16682024-09-06T06:38:51.946ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92551 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62219 req_id = 668038eb-ac70-487e-b52c-68076678feb0 response_code = 204 uri = /switch-ports
16692024-09-06T06:38:51.950ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3921 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62219 req_id = 8d45af0c-8a44-4847-9d09-448a240ff5e9 response_code = 200 uri = /network-bootstore-config
16702024-09-06T06:38:52.311ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58459
16712024-09-06T06:38:52.311ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58459 req_id = c178a65d-ff57-498c-9403-6afad7a9d17f response_code = 200 uri = /v2p
16722024-09-06T06:38:52.460ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58796
16732024-09-06T06:38:52.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58796 req_id = 25a1888c-5b95-4f15-9b5c-14e6eb83a2ab response_code = 200 uri = /vpc-routes
16742024-09-06T06:38:52.467ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43438
16752024-09-06T06:38:52.467ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43438 req_id = a370d730-37b1-4727-9f47-ab58c9926207 response_code = 200 uri = /vpc-routes
16762024-09-06T06:38:52.475ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58796 req_id = dc316dea-4ef9-43aa-a509-602e33318fca response_code = 204 uri = /vpc-routes
16772024-09-06T06:38:52.483ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7661 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 6a174c28-e532-4184-aea8-69f4b107f4b0 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
16782024-09-06T06:38:52.484ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43438 req_id = 3c42e7ef-0278-4eff-a948-b6c17591c3af response_code = 204 uri = /vpc-routes
16792024-09-06T06:38:52.579ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49955
16802024-09-06T06:38:52.579ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16812024-09-06T06:38:52.616ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16822024-09-06T06:38:52.672ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93128 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49955 req_id = 792d7443-b0d7-405c-96c5-fb869dc73c73 response_code = 204 uri = /switch-ports
16832024-09-06T06:38:52.672ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45784
16842024-09-06T06:38:52.672ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51587
16852024-09-06T06:38:52.673ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
16862024-09-06T06:38:52.676ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3357 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51587 req_id = 50e35ced-0330-4360-9425-d3642c634f84 response_code = 200 uri = /v2p
16872024-09-06T06:38:52.676ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3482 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49955 req_id = 80b93e62-14ed-4e57-a77f-9e9c9e7099a4 response_code = 200 uri = /network-bootstore-config
16882024-09-06T06:38:52.707ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
16892024-09-06T06:38:52.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98470 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45784 req_id = e3fe4646-7e9a-405f-95bc-4a2838d8a493 response_code = 204 uri = /switch-ports
16902024-09-06T06:38:52.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45784 req_id = 7f718bd2-00d8-4867-92c0-256d54913d85 response_code = 200 uri = /network-bootstore-config
16912024-09-06T06:39:14.303ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52831
16922024-09-06T06:39:14.303ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 36 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52831 req_id = f6010315-12f0-4964-9a3c-2d50f4da0b19 response_code = 200 uri = /baseboard
16932024-09-06T06:39:21.361ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51892
16942024-09-06T06:39:21.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51892 req_id = c0163d3f-96a4-4beb-a16a-0094d8c9b790 response_code = 200 uri = /vpc-routes
16952024-09-06T06:39:21.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51892 req_id = 8d22a404-01ba-4219-9f56-8ca1f6d49b7b response_code = 204 uri = /vpc-routes
16962024-09-06T06:39:21.451ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42078
16972024-09-06T06:39:21.451ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42078 req_id = 0fa32d69-9bf4-46c5-bb0c-ff23eb234f43 response_code = 200 uri = /v2p
16982024-09-06T06:39:21.453ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48166
16992024-09-06T06:39:21.453ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17002024-09-06T06:39:21.488ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17012024-09-06T06:39:21.546ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92648 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48166 req_id = 5a79f70e-4ec1-420d-8ba9-a6befd7525b3 response_code = 204 uri = /switch-ports
17022024-09-06T06:39:21.546ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48166 req_id = 737dd692-158c-4c2d-8c7d-d5c6ae3d6461 response_code = 200 uri = /network-bootstore-config
17032024-09-06T06:39:22.187ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55860
17042024-09-06T06:39:22.187ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55860 req_id = 028fe764-f1ad-4304-a6da-8accd3aa59f7 response_code = 200 uri = /vpc-routes
17052024-09-06T06:39:22.198ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55860 req_id = 6c7d921e-509a-470b-9f2d-315818fe234d response_code = 204 uri = /vpc-routes
17062024-09-06T06:39:22.270ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40212
17072024-09-06T06:39:22.270ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 653 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40212 req_id = 103bef4b-aa7f-4616-bc2e-17b075a30cd3 response_code = 200 uri = /vpc-routes
17082024-09-06T06:39:22.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40212 req_id = 0ffd9c35-dfff-429b-9665-247c77ec1770 response_code = 204 uri = /vpc-routes
17092024-09-06T06:39:22.326ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65281
17102024-09-06T06:39:22.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 858 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65281 req_id = 3ce0fedf-c01c-4ece-9166-662d6e7388ce response_code = 200 uri = /v2p
17112024-09-06T06:39:22.335ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54694
17122024-09-06T06:39:22.335ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17132024-09-06T06:39:22.369ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17142024-09-06T06:39:22.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93420 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:54694 req_id = 3aff3c62-e5df-4d43-b47c-64b19609239a response_code = 204 uri = /switch-ports
17152024-09-06T06:39:22.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51356
17162024-09-06T06:39:22.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59991
17172024-09-06T06:39:22.429ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17182024-09-06T06:39:22.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 284 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54694 req_id = 659f81df-32c9-4e2a-a838-4dbf0dd522bc response_code = 200 uri = /network-bootstore-config
17192024-09-06T06:39:22.433ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3541 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59991 req_id = 87a375b0-400f-4ac6-84e3-b4d4a1abe2d0 response_code = 200 uri = /v2p
17202024-09-06T06:39:22.464ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17212024-09-06T06:39:22.484ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6845 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 89590866-8609-4912-ab8a-61ae3b6ea05c response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
17222024-09-06T06:39:22.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90300 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51356 req_id = 3289ebe1-c751-484f-b0b5-f3f96182f39d response_code = 204 uri = /switch-ports
17232024-09-06T06:39:22.520ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 617 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51356 req_id = ba118cf9-4570-41e5-b8fc-e4f625b520cd response_code = 200 uri = /network-bootstore-config
17242024-09-06T06:39:44.342ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42147
17252024-09-06T06:39:44.342ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42147 req_id = b06be60d-ef41-4409-ad26-5dda80566e44 response_code = 200 uri = /baseboard
17262024-09-06T06:39:51.557ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59251
17272024-09-06T06:39:51.557ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59251 req_id = c202701a-caf4-4fd9-b9ee-671df2373a5c response_code = 200 uri = /vpc-routes
17282024-09-06T06:39:51.567ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59251 req_id = 827a4215-82a7-4441-80ba-d9a8770772ab response_code = 204 uri = /vpc-routes
17292024-09-06T06:39:51.588ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40658
17302024-09-06T06:39:51.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40658 req_id = 55edae63-f320-4de2-ba58-8329d5e8562d response_code = 200 uri = /v2p
17312024-09-06T06:39:51.801ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53332
17322024-09-06T06:39:51.801ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17332024-09-06T06:39:51.839ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17342024-09-06T06:39:51.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105261 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:53332 req_id = eaa66a0d-01fd-4731-8ee7-860c5aed2246 response_code = 204 uri = /switch-ports
17352024-09-06T06:39:51.907ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53332 req_id = 3a61e73f-d534-4c24-ba0a-8b4525317b63 response_code = 200 uri = /network-bootstore-config
17362024-09-06T06:39:52.283ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58335
17372024-09-06T06:39:52.283ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58335 req_id = b9ad0b10-f85f-4a0a-932c-02b63cb9646b response_code = 200 uri = /vpc-routes
17382024-09-06T06:39:52.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58335 req_id = 48e05662-2891-4f92-b126-d6481af40fea response_code = 204 uri = /vpc-routes
17392024-09-06T06:39:52.335ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62847
17402024-09-06T06:39:52.335ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62847 req_id = be82773c-c724-48d3-a044-329d4bde49a2 response_code = 200 uri = /vpc-routes
17412024-09-06T06:39:52.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1192 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62847 req_id = ca0f50bc-f078-4d5f-9a91-5d636de8aac6 response_code = 204 uri = /vpc-routes
17422024-09-06T06:39:52.427ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64848
17432024-09-06T06:39:52.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64848 req_id = ed892f95-8901-4c94-b3bf-8e4d5867f384 response_code = 200 uri = /v2p
17442024-09-06T06:39:52.454ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61898
17452024-09-06T06:39:52.454ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17462024-09-06T06:39:52.492ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17472024-09-06T06:39:52.546ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91929 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61898 req_id = 8bd7ff54-68e6-4477-836f-c26ae31770ad response_code = 204 uri = /switch-ports
17482024-09-06T06:39:52.552ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5663 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 4a76920a-ec63-49f6-95ef-6cc532fa2b11 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
17492024-09-06T06:39:52.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61898 req_id = 70ffc9f3-fc53-41a7-a11d-295d853d00f4 response_code = 200 uri = /network-bootstore-config
17502024-09-06T06:39:52.616ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39345
17512024-09-06T06:39:52.617ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 842 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39345 req_id = 788024dd-58a6-4bf2-827d-a8795db5b8d5 response_code = 200 uri = /v2p
17522024-09-06T06:39:52.736ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32804
17532024-09-06T06:39:52.736ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17542024-09-06T06:39:52.776ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17552024-09-06T06:39:52.830ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93408 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:32804 req_id = 30ff936b-c6a3-4314-8915-eb755cd1a859 response_code = 204 uri = /switch-ports
17562024-09-06T06:39:52.830ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:32804 req_id = c518b972-b66e-4df9-9239-600494963e3c response_code = 200 uri = /network-bootstore-config
17572024-09-06T06:40:14.380ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55492
17582024-09-06T06:40:14.380ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 497 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55492 req_id = 130e01a1-abae-433a-a08a-c9b1ccb9a38c response_code = 200 uri = /baseboard
17592024-09-06T06:40:21.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47539
17602024-09-06T06:40:21.322ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47539 req_id = 1e3ea1fc-7b81-4d25-afac-d860c50c425c response_code = 200 uri = /vpc-routes
17612024-09-06T06:40:21.338ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 930 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47539 req_id = 4f04b13a-61c8-4a5a-aa3c-271863424f01 response_code = 204 uri = /vpc-routes
17622024-09-06T06:40:21.530ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49137
17632024-09-06T06:40:21.530ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17642024-09-06T06:40:21.568ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17652024-09-06T06:40:21.624ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93716 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49137 req_id = e6bea717-6c9e-4743-8d34-85f1534bd677 response_code = 204 uri = /switch-ports
17662024-09-06T06:40:21.624ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35704
17672024-09-06T06:40:21.624ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35704 req_id = 58bf3bd0-c864-4a30-97a8-9325a921835c response_code = 200 uri = /v2p
17682024-09-06T06:40:21.624ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49137 req_id = 3c6814cf-295f-4367-b47d-394a78518dcc response_code = 200 uri = /network-bootstore-config
17692024-09-06T06:40:22.243ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63147
17702024-09-06T06:40:22.244ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1015 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63147 req_id = f57cf1c0-4cef-4d64-be60-6d32ee1b6dbe response_code = 200 uri = /vpc-routes
17712024-09-06T06:40:22.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63147 req_id = 45f568dc-1304-4477-a491-13e224dede7f response_code = 204 uri = /vpc-routes
17722024-09-06T06:40:22.298ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57667
17732024-09-06T06:40:22.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57667 req_id = d5f0b0c4-96a9-4be8-83a9-5593cb28c507 response_code = 200 uri = /v2p
17742024-09-06T06:40:22.319ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42806
17752024-09-06T06:40:22.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42806 req_id = 4ed7efa4-189d-4b5c-8027-6a8647df4817 response_code = 200 uri = /v2p
17762024-09-06T06:40:22.320ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46602
17772024-09-06T06:40:22.320ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46602 req_id = 731c0041-e0c6-4d31-9369-9af1658fd99f response_code = 200 uri = /vpc-routes
17782024-09-06T06:40:22.328ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46602 req_id = 026136bb-eb30-4870-8e75-6115a220508e response_code = 204 uri = /vpc-routes
17792024-09-06T06:40:22.382ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45898
17802024-09-06T06:40:22.382ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17812024-09-06T06:40:22.420ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17822024-09-06T06:40:22.472ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90397 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45898 req_id = b0f62e63-1abf-48f5-838b-3da60f4c1f89 response_code = 204 uri = /switch-ports
17832024-09-06T06:40:22.472ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53321
17842024-09-06T06:40:22.472ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17852024-09-06T06:40:22.476ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3943 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45898 req_id = 1e55ab48-8c9b-4384-8e83-ba0f4be9d31a response_code = 200 uri = /network-bootstore-config
17862024-09-06T06:40:22.483ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6616 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = fb649aa3-e880-4a23-aa0f-4faef21f2728 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
17872024-09-06T06:40:22.508ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17882024-09-06T06:40:22.560ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88093 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53321 req_id = f881591b-76e9-4332-988b-0667e0f59a0e response_code = 204 uri = /switch-ports
17892024-09-06T06:40:22.561ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 599 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53321 req_id = 01e49ca1-71a7-45a2-8737-8cd99786c6b4 response_code = 200 uri = /network-bootstore-config
17902024-09-06T06:40:44.418ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48421
17912024-09-06T06:40:44.419ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 674 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48421 req_id = 1c6f2039-6249-4729-a371-4ded9ec45fa7 response_code = 200 uri = /baseboard
17922024-09-06T06:40:51.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32986
17932024-09-06T06:40:51.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32986 req_id = fbc6976c-d21d-4c9c-9ab2-8a6fb54b4e6c response_code = 200 uri = /vpc-routes
17942024-09-06T06:40:51.542ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:32986 req_id = 9cbe46db-7309-41fb-b496-d45f11be1073 response_code = 204 uri = /vpc-routes
17952024-09-06T06:40:51.819ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49145
17962024-09-06T06:40:51.820ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 153 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49145 req_id = f2ed1e66-ba4f-46a5-b366-5de1d66dde67 response_code = 200 uri = /v2p
17972024-09-06T06:40:51.935ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36292
17982024-09-06T06:40:51.935ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17992024-09-06T06:40:51.978ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18002024-09-06T06:40:52.062ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126513 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36292 req_id = 124b32dd-014a-4305-a033-d005d79e7ed0 response_code = 204 uri = /switch-ports
18012024-09-06T06:40:52.062ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36292 req_id = 5efa5c50-c81b-4151-8fcf-a60be28fde67 response_code = 200 uri = /network-bootstore-config
18022024-09-06T06:40:52.141ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37862
18032024-09-06T06:40:52.142ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18042024-09-06T06:40:52.142ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18052024-09-06T06:40:52.142ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18062024-09-06T06:40:52.142ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18072024-09-06T06:40:52.142ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
18082024-09-06T06:40:52.142ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18092024-09-06T06:40:52.142ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18102024-09-06T06:40:52.142ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 846 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37862 req_id = 04d94ab0-89f8-4dd8-9704-67798b385306 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18112024-09-06T06:40:52.247ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40136
18122024-09-06T06:40:52.248ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40136 req_id = 0bedfdf6-a1b7-45fe-9ed3-689208094a0a response_code = 200 uri = /vpc-routes
18132024-09-06T06:40:52.257ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40136 req_id = c7533172-f4e9-4e44-9d1c-2d73cad1e910 response_code = 204 uri = /vpc-routes
18142024-09-06T06:40:52.386ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38473
18152024-09-06T06:40:52.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38473 req_id = b5f2ecc2-48b4-4437-9eb3-24ea7ac4168f response_code = 200 uri = /v2p
18162024-09-06T06:40:52.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58254
18172024-09-06T06:40:52.419ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18182024-09-06T06:40:52.459ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18192024-09-06T06:40:52.513ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94342 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58254 req_id = 4578d19d-7bae-4be5-bcd3-4ef057dcfd2c response_code = 204 uri = /switch-ports
18202024-09-06T06:40:52.520ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6502 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 4382151d-4df3-4911-8530-7c39a8b88e5b response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
18212024-09-06T06:40:52.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58254 req_id = 7b2c1328-8fe4-43ac-85cb-773373d5964f response_code = 200 uri = /network-bootstore-config
18222024-09-06T06:40:52.571ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60095
18232024-09-06T06:40:52.572ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 966 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60095 req_id = d364c3ba-8b21-4ce6-b2d9-5ff311a6ed77 response_code = 200 uri = /vpc-routes
18242024-09-06T06:40:52.584ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 172 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60095 req_id = fe50adde-7728-461c-8c72-194aab8feed1 response_code = 204 uri = /vpc-routes
18252024-09-06T06:40:52.666ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50373
18262024-09-06T06:40:52.668ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18272024-09-06T06:40:52.713ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18282024-09-06T06:40:52.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104662 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50373 req_id = 81b24735-7e16-41ee-bd5e-7bee77cd37e5 response_code = 204 uri = /switch-ports
18292024-09-06T06:40:52.770ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64187
18302024-09-06T06:40:52.770ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49877
18312024-09-06T06:40:52.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50373 req_id = 5f692ae1-dc1f-4e92-86da-de1626566739 response_code = 200 uri = /network-bootstore-config
18322024-09-06T06:40:52.771ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18332024-09-06T06:40:52.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 958 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64187 req_id = de2ae4ff-c560-4ff2-a436-d89814a4eb9f response_code = 200 uri = /v2p
18342024-09-06T06:40:52.771ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18352024-09-06T06:40:52.771ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18362024-09-06T06:40:52.771ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18372024-09-06T06:40:52.771ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
18382024-09-06T06:40:52.771ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18392024-09-06T06:40:52.772ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18402024-09-06T06:40:52.772ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 984 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49877 req_id = 5f0c2676-3a19-4ec8-ae52-35382bee0970 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18412024-09-06T06:40:52.878ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61549
18422024-09-06T06:40:52.879ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9ae56977-4835-4c3f-ab36-610dae081d5c, kind: Service { id: 81c3dd8c-a34f-489a-91c0-599e726f0b45 }, name: Name("external-dns-81c3dd8c-a34f-489a-91c0-599e726f0b45"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 227])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c76b0ebb-9714-464b-887a-8b5d03d9c8fd, kind: Service { id: bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d }, name: Name("external-dns-bbb8d6c8-3b06-44b1-a1b9-e4dbc8390f5d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0c4b7a66-0f06-420b-9422-6d4607c5823c, kind: Service { id: f4c090fd-0141-407e-a547-8163f49da46c }, name: Name("nexus-f4c090fd-0141-407e-a547-8163f49da46c"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 252, 19])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7d5ba05c-32f6-4b31-b1fb-56de407e3a6b, kind: Service { id: e9c280ee-8313-41ab-8fa4-df346c954189 }, name: Name("nexus-e9c280ee-8313-41ab-8fa4-df346c954189"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e06d59cb-3fd1-4320-a97d-c5eb06976fbf, kind: Service { id: 5d85f2d3-87bb-4fce-a9bf-1f1805221476 }, name: Name("nexus-5d85f2d3-87bb-4fce-a9bf-1f1805221476"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18432024-09-06T06:40:52.879ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18442024-09-06T06:40:52.879ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18452024-09-06T06:40:52.879ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18462024-09-06T06:40:52.879ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
18472024-09-06T06:40:52.879ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18482024-09-06T06:40:52.879ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18492024-09-06T06:40:52.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 907 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61549 req_id = f133e811-e3d9-4bcc-a7f2-7cf5f4033511 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18502024-09-06T06:41:14.452ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53066
18512024-09-06T06:41:14.452ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 616 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53066 req_id = 48eba3b5-22f7-4449-b97c-e841d2f9c037 response_code = 200 uri = /baseboard
18522024-09-06T06:41:21.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38135
18532024-09-06T06:41:21.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38135 req_id = af42bf32-86e5-42b1-bd63-9ad69f43f4bd response_code = 200 uri = /vpc-routes
18542024-09-06T06:41:21.479ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41565
18552024-09-06T06:41:21.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 161 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41565 req_id = a7912236-147d-4ecb-9e84-196e27d24fc7 response_code = 200 uri = /v2p
18562024-09-06T06:41:21.483ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38135 req_id = ca2b054f-de23-45f8-8b01-8c08ae5e410a response_code = 204 uri = /vpc-routes
18572024-09-06T06:41:21.529ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49606
18582024-09-06T06:41:21.529ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18592024-09-06T06:41:21.567ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18602024-09-06T06:41:21.625ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95616 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49606 req_id = 1932f43c-02f8-4abc-823e-cf8487bede7f response_code = 204 uri = /switch-ports
18612024-09-06T06:41:21.625ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49606 req_id = 4eadc991-a8b9-4311-b1e8-bfdbe7087c4a response_code = 200 uri = /network-bootstore-config
18622024-09-06T06:41:22.156ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47527
18632024-09-06T06:41:22.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47527 req_id = 1db37214-618b-4995-9070-1790ba357bf2 response_code = 200 uri = /vpc-routes
18642024-09-06T06:41:22.168ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1065 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47527 req_id = 29cb23e7-555f-49a1-a616-0237ebc852ee response_code = 204 uri = /vpc-routes
18652024-09-06T06:41:22.254ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48961
18662024-09-06T06:41:22.254ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48961 req_id = 52777b63-5e3f-4a9a-9b78-a124fc0d141b response_code = 200 uri = /v2p
18672024-09-06T06:41:22.329ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43008
18682024-09-06T06:41:22.329ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18692024-09-06T06:41:22.370ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18702024-09-06T06:41:22.434ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105312 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43008 req_id = f94c45b8-c518-43fe-bf2f-89f77a938c0f response_code = 204 uri = /switch-ports
18712024-09-06T06:41:22.434ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35314
18722024-09-06T06:41:22.434ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36161
18732024-09-06T06:41:22.435ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18742024-09-06T06:41:22.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3845 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43008 req_id = 83427e68-af05-4fc1-972c-09607365d71d response_code = 200 uri = /network-bootstore-config
18752024-09-06T06:41:22.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3806 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36161 req_id = b9accb5f-1e29-4b72-8389-48d35bad5217 response_code = 200 uri = /vpc-routes
18762024-09-06T06:41:22.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54505
18772024-09-06T06:41:22.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54505 req_id = 7216b47b-6f30-4f78-90b4-19b264f20cf2 response_code = 200 uri = /v2p
18782024-09-06T06:41:22.455ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36161 req_id = 16a57a64-b1f9-4600-b6bc-b41037374dba response_code = 204 uri = /vpc-routes
18792024-09-06T06:41:22.476ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18802024-09-06T06:41:22.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101842 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35314 req_id = 4c91fca9-2fae-4a04-b768-86439ec1d132 response_code = 204 uri = /switch-ports
18812024-09-06T06:41:22.537ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35314 req_id = 23b6e20e-64c2-491e-979b-ec7a97b80bb0 response_code = 200 uri = /network-bootstore-config
18822024-09-06T06:41:22.671ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6893 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 04d55db7-2b9d-4bab-842a-748f23b6bffb response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
18832024-09-06T06:41:44.492ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42845
18842024-09-06T06:41:44.492ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 724 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42845 req_id = 041fcd69-7dbf-406d-9ade-62603e72e951 response_code = 200 uri = /baseboard
18852024-09-06T06:41:51.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41892
18862024-09-06T06:41:51.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 783 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41892 req_id = 3d3150ba-4b87-4ed1-909f-4c175f1126fa response_code = 200 uri = /vpc-routes
18872024-09-06T06:41:51.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41892 req_id = f5788fe6-a37c-4711-918c-cf8f49c00b41 response_code = 204 uri = /vpc-routes
18882024-09-06T06:41:51.830ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62788
18892024-09-06T06:41:51.831ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62788 req_id = 16545d96-ac9a-4247-b8ef-263c91983ff8 response_code = 200 uri = /v2p
18902024-09-06T06:41:51.862ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60290
18912024-09-06T06:41:51.862ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18922024-09-06T06:41:51.895ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18932024-09-06T06:41:51.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85933 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60290 req_id = 7f8d6a1a-2031-4de2-8d27-3e99cbd10c1d response_code = 204 uri = /switch-ports
18942024-09-06T06:41:51.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60290 req_id = 0d96962f-a7d4-427a-a5de-cadadf41e847 response_code = 200 uri = /network-bootstore-config
18952024-09-06T06:41:52.294ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55045
18962024-09-06T06:41:52.294ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55045 req_id = f4998ce1-3946-47b8-b278-ab1008bae8b1 response_code = 200 uri = /vpc-routes
18972024-09-06T06:41:52.397ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55708
18982024-09-06T06:41:52.398ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1077 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55708 req_id = 7aa9ed62-6341-4d94-bd10-bab52975d7ab response_code = 200 uri = /v2p
18992024-09-06T06:41:52.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55045 req_id = 10e8c01a-5a9d-4b6e-8bef-4e31efb352d3 response_code = 204 uri = /vpc-routes
19002024-09-06T06:41:52.436ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44155
19012024-09-06T06:41:52.436ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44155 req_id = 579107e7-ba56-4e7b-84dc-e9ef1177b363 response_code = 200 uri = /vpc-routes
19022024-09-06T06:41:52.458ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40064
19032024-09-06T06:41:52.458ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40064 req_id = 8db84ab6-a7a2-42f2-abc0-5e06d5c753c8 response_code = 200 uri = /v2p
19042024-09-06T06:41:52.468ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44155 req_id = 971ca33f-1903-4d9d-b3a3-9a20841eb270 response_code = 204 uri = /vpc-routes
19052024-09-06T06:41:52.515ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48868
19062024-09-06T06:41:52.515ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19072024-09-06T06:41:52.556ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19082024-09-06T06:41:52.613ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98178 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48868 req_id = 7183d349-c0ff-47b7-9fc6-eb3228c57ece response_code = 204 uri = /switch-ports
19092024-09-06T06:41:52.613ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51240
19102024-09-06T06:41:52.614ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19112024-09-06T06:41:52.649ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19122024-09-06T06:41:52.702ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88833 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51240 req_id = 8f78998b-6668-4c7d-9e64-85c7facae28e response_code = 204 uri = /switch-ports
19132024-09-06T06:41:52.703ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 815 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48868 req_id = 3a708916-6a8b-49e1-b8b3-15b3efbd9627 response_code = 200 uri = /network-bootstore-config
19142024-09-06T06:41:52.703ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 841 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51240 req_id = 1c81eb68-2ba5-4de6-b14a-d4a5c6b1fefa response_code = 200 uri = /network-bootstore-config
19152024-09-06T06:41:52.710ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7206 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = 3cf1502e-f850-4871-bf0c-49d09563a999 response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
19162024-09-06T06:42:14.525ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57395
19172024-09-06T06:42:14.525ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57395 req_id = 00c58172-32d2-4613-944a-3053041c35ed response_code = 200 uri = /baseboard
19182024-09-06T06:42:19.001ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36255
19192024-09-06T06:42:19.001ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 732 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36255 req_id = 2cdedfe6-8e72-44ac-b058-763a1f2cbfd0 response_code = 200 uri = /v2p
19202024-09-06T06:42:19.002ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(14029028) }
19212024-09-06T06:42:19.002ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36255 req_id = bbacaffa-872a-4088-bc47-f99902bd96fb response_code = 204 uri = /v2p
19222024-09-06T06:42:19.177ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63831
19232024-09-06T06:42:19.178ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:570 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(14029028))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-6a735a8f656", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 4a9b558c-89c1-4e2d-8074-1b218f5671ef, read_only_parent: Some(Volume { block_size: 512, id: 54ab55fb-a25f-4ad8-817e-27d27bd7de75, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 9b3a1487-de59-4ca7-a8e6-dc31a98fe5fb, key: Some("6x/gVlbTN+Bv0b+w9vmOqGm+C2sFTSaatC/D+sjfSL4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::15]:19001", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::12]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4a9b558c-89c1-4e2d-8074-1b218f5671ef, key: Some("hnLnl80cw1+FZtnSTiFoRBfY9jmqCoi3b4O10Zfcncs="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19003", "[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::16]:19000"] } }] } }], cloud_init_bytes: Some(..) } instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c instance_runtime = InstanceRuntimeState { propolis_id: Some(3ff7c611-2e3c-453a-b476-a13af9fea981 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-09-06T06:42:18.607570614Z } metadata = InstanceMetadata { silo_id: b4c2f952-f953-455d-9604-efc053a121ee, project_id: 088ccfb0-42d8-4ea8-941a-6b6b6a586936 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-06T06:42:19.171938105Z }
19242024-09-06T06:42:19.179ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:603 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19252024-09-06T06:42:19.179ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1008 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c migration_id = None propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(14029028))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46d4cb3a-3dce-43fe-bc57-e50edf04b512, kind: Instance { id: b3df2a2c-7c6c-4735-9286-8c631ebabc6c }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 115, 49])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14029028), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-6a735a8f656", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 4a9b558c-89c1-4e2d-8074-1b218f5671ef, read_only_parent: Some(Volume { block_size: 512, id: 54ab55fb-a25f-4ad8-817e-27d27bd7de75, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 9b3a1487-de59-4ca7-a8e6-dc31a98fe5fb, key: Some("6x/gVlbTN+Bv0b+w9vmOqGm+C2sFTSaatC/D+sjfSL4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::15]:19001", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::12]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4a9b558c-89c1-4e2d-8074-1b218f5671ef, key: Some("hnLnl80cw1+FZtnSTiFoRBfY9jmqCoi3b4O10Zfcncs="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19003", "[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::16]:19000"] } }] } }], cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-06T06:42:19.171938105Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
19262024-09-06T06:42:19.179ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1986 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63831 req_id = 1aa6c2cb-b904-4cb3-b4d8-0eddf885989e response_code = 200 uri = /vmms/3ff7c611-2e3c-453a-b476-a13af9fea981
19272024-09-06T06:42:19.331ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56937
19282024-09-06T06:42:19.331ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56937 req_id = b25f40a8-ec12-4918-a44b-5b74d7c7bc01 response_code = 200 uri = /vmms/3ff7c611-2e3c-453a-b476-a13af9fea981/state
19292024-09-06T06:42:19.332ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 246, 115, 49]), slot: 0, vni: Vni { inner: 14029028 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
19302024-09-06T06:42:19.332ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
19312024-09-06T06:42:19.370ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981 file = illumos-utils/src/zone.rs:316 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19322024-09-06T06:42:19.392ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981 file = illumos-utils/src/zone.rs:354 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19332024-09-06T06:42:20.386ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19342024-09-06T06:42:20.386ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981 zone file = illumos-utils/src/running_zone.rs:488 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19352024-09-06T06:42:20.571ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 48.10105ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19362024-09-06T06:42:20.629ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 91.338228ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19372024-09-06T06:42:20.737ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 189.871962ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19382024-09-06T06:42:20.944ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 518.956697ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19392024-09-06T06:42:21.321ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39541
19402024-09-06T06:42:21.321ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39541 req_id = 72b9b3a5-ab6f-4e01-af59-eddb13c280a9 response_code = 200 uri = /vpc-routes
19412024-09-06T06:42:21.478ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 458.753597ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19422024-09-06T06:42:21.542ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62755
19432024-09-06T06:42:21.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 331 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39541 req_id = 28e19369-4d28-475a-bd49-4e98bb921cd3 response_code = 204 uri = /vpc-routes
19442024-09-06T06:42:21.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56587
19452024-09-06T06:42:21.609ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19462024-09-06T06:42:21.664ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19472024-09-06T06:42:21.743ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 133505 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56587 req_id = 76ba95e0-eb23-4a71-adb0-8f52f3eb1b66 response_code = 204 uri = /switch-ports
19482024-09-06T06:42:21.743ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36859
19492024-09-06T06:42:21.743ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 215 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36859 req_id = 0a9cb033-d187-4c0e-8780-1d1d78dce8ec response_code = 200 uri = /v2p
19502024-09-06T06:42:21.744ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1073 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56587 req_id = 125b5363-0300-4932-83e9-596a9cd835a7 response_code = 200 uri = /network-bootstore-config
19512024-09-06T06:42:21.948ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 655.592238ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19522024-09-06T06:42:22.153ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47641
19532024-09-06T06:42:22.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47641 req_id = c98c3268-e8da-42c0-aaf3-0371c8257c87 response_code = 200 uri = /v2p
19542024-09-06T06:42:22.214ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46660
19552024-09-06T06:42:22.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64096
19562024-09-06T06:42:22.257ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64096 req_id = 6da55564-4982-41eb-9b4d-6fb9ffabdeaa response_code = 200 uri = /vpc-routes
19572024-09-06T06:42:22.289ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65033
19582024-09-06T06:42:22.289ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65033 req_id = de6b875f-d1e0-4620-835c-bd552b38628b response_code = 200 uri = /vpc-routes
19592024-09-06T06:42:22.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:65033 req_id = 1a21b6a1-9909-44a3-8d4e-ca7da063a584 response_code = 204 uri = /vpc-routes
19602024-09-06T06:42:22.319ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46155
19612024-09-06T06:42:22.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64096 req_id = d8de4f63-b5fe-4a86-9cfc-b64cfe883425 response_code = 204 uri = /vpc-routes
19622024-09-06T06:42:22.348ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47852
19632024-09-06T06:42:22.348ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19642024-09-06T06:42:22.391ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19652024-09-06T06:42:22.454ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106100 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47852 req_id = b219b0e6-bb69-42d2-b521-cc6dbe72b4da response_code = 204 uri = /switch-ports
19662024-09-06T06:42:22.454ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47852 req_id = be831f16-45be-47c2-9241-3c701251e118 response_code = 200 uri = /network-bootstore-config
19672024-09-06T06:42:22.474ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36616
19682024-09-06T06:42:22.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36616 req_id = 53db5e20-0228-4865-9d32-12499bec69a8 response_code = 200 uri = /v2p
19692024-09-06T06:42:22.489ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37810
19702024-09-06T06:42:22.489ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19712024-09-06T06:42:22.528ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19722024-09-06T06:42:22.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98440 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:37810 req_id = 39e7e067-0b7c-42d9-abf6-5454e21b4f70 response_code = 204 uri = /switch-ports
19732024-09-06T06:42:22.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37810 req_id = d811fc73-d55a-42d7-b340-5c1b6df70ae9 response_code = 200 uri = /network-bootstore-config
19742024-09-06T06:42:22.615ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 1.205381853s file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19752024-09-06T06:42:22.669ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5691 local_addr = [fd00:1122:3344:101::1]:38689 method = GET remote_addr = [fd00:1122:3344:101::d]:56516 req_id = a46232f2-ff87-4f81-84ca-e9af3763a19c response_code = 200 sled_id = 58853ca1-8d5a-4912-87e6-57de713a602c uri = /58853ca1-8d5a-4912-87e6-57de713a602c
19762024-09-06T06:42:23.832ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 522.819777ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19772024-09-06T06:42:24.367ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 1.315295172s file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 zone = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
19782024-09-06T06:42:25.710ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981 file = sled-agent/src/instance.rs:1466 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19792024-09-06T06:42:25.721ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 43.835858ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19802024-09-06T06:42:25.777ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 126.97263ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19812024-09-06T06:42:25.914ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 197.139902ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19822024-09-06T06:42:26.121ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981") failed: Property not found. retry in 426.887285ms file = illumos-utils/src/svc.rs:36 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19832024-09-06T06:42:26.558ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1475 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19842024-09-06T06:42:26.595ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1499 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19852024-09-06T06:42:26.741ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:744 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19862024-09-06T06:42:26.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5200042 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62755 req_id = 63c4f7d5-e9eb-4020-aa3f-94cd14109f81 response_code = 200 uri = /vmms/3ff7c611-2e3c-453a-b476-a13af9fea981/state
19872024-09-06T06:42:26.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4420901 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46155 req_id = 0d48d64e-214f-4a32-a108-f6332646e2cc response_code = 200 uri = /vmms/3ff7c611-2e3c-453a-b476-a13af9fea981/state
19882024-09-06T06:42:26.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4527964 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46660 req_id = 64d7909a-363d-4ca2-ab31-ce8e3312c9f4 response_code = 200 uri = /vmms/3ff7c611-2e3c-453a-b476-a13af9fea981/state
19892024-09-06T06:42:26.743ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-09-06T06:42:26.743487554Z } file = sled-agent/src/instance.rs:594 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19902024-09-06T06:42:26.743ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-06T06:42:26.743487554Z } propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19912024-09-06T06:42:26.743ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-06T06:42:26.743487554Z }, migration_in: None, migration_out: None }
19922024-09-06T06:42:26.790ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-09-06T06:42:26.790365183Z } file = sled-agent/src/instance.rs:594 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19932024-09-06T06:42:26.790ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-06T06:42:26.790365183Z } propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19942024-09-06T06:42:26.790ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-06T06:42:26.790365183Z }, migration_in: None, migration_out: None }
19952024-09-06T06:42:36.559ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61289
19962024-09-06T06:42:36.560ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 595 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61289 req_id = 9794f996-87c4-43e9-9c4a-bd55f8b67527 response_code = 200 uri = /vmms/3ff7c611-2e3c-453a-b476-a13af9fea981/state
19972024-09-06T06:42:36.560ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-09-06T06:42:36.560321449Z } file = sled-agent/src/instance.rs:594 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19982024-09-06T06:42:36.560ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-06T06:42:36.560321449Z } propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
19992024-09-06T06:42:36.560ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-06T06:42:36.560321449Z }, migration_in: None, migration_out: None }
20002024-09-06T06:42:36.576ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-09-06T06:42:36.576085094Z } file = sled-agent/src/instance.rs:594 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
20012024-09-06T06:42:36.576ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-09-06T06:42:36.576085094Z } propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
20022024-09-06T06:42:36.576ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:630 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
20032024-09-06T06:42:36.576ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20042024-09-06T06:42:36.577ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20052024-09-06T06:42:36.577ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/acfe4e3e-8cb9-4336-9553-a7677bbf6061/debug/bundle/zone", "/pool/int/bbad1fce-0f63-461f-8b49-dd0c7a492808/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/7516b8f4-123a-4b07-8c44-30d87eec7e9c/crypt/debug", "/pool/ext/56647842-7219-46e1-9978-bdffa4d4e8cf/crypt/debug", "/pool/ext/7a42b808-7b1f-406b-87b7-e6aeecc73464/crypt/debug", "/pool/ext/a32e2a10-4360-4704-b36b-dc63733b1b01/crypt/debug", "/pool/ext/f881a8f5-60a1-4050-a3bb-fd18022091f4/crypt/debug", "/pool/ext/681fcf67-79af-4c28-8e2f-62c72e00e09a/crypt/debug", "/pool/ext/97ec7cac-82ac-4161-b909-856eea5d7741/crypt/debug", "/pool/ext/9d6ee5ce-b9f0-4615-ad18-41313f80d200/crypt/debug", "/pool/ext/a433fad0-5ca4-4fce-ac22-ce488e17cfe3/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981
20062024-09-06T06:42:37.236ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981", bundle_id: d6bb1ea2-fb4c-48ec-a510-6dce1deffb37 }, time_created: 2024-09-06T06:42:36.578045462Z, version: 0, cause: TerminatedInstance }
20072024-09-06T06:42:37.236ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_3ff7c611-2e3c-453a-b476-a13af9fea981 file = sled-agent/src/instance.rs:843 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
20082024-09-06T06:42:39.000ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
20092024-09-06T06:42:39.017ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(14029028), kind: System }
20102024-09-06T06:42:39.017ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(14029028), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
20112024-09-06T06:42:39.017ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-06T06:42:39.017863411Z }, migration_in: None, migration_out: None }
20122024-09-06T06:42:39.097ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-06T06:42:39.017863411Z }, migration_in: None, migration_out: None }
20132024-09-06T06:42:39.101ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:780 instance_id = b3df2a2c-7c6c-4735-9286-8c631ebabc6c propolis_id = 3ff7c611-2e3c-453a-b476-a13af9fea981
20142024-09-06T06:42:39.584ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42528
20152024-09-06T06:42:39.584ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42528 req_id = fa3338de-506d-4aea-9946-a4cad36576cd response_code = 200 uri = /vpc-routes
20162024-09-06T06:42:39.593ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:42528 req_id = fc768416-60d1-4ccd-9ba3-966fa69389e0 response_code = 204 uri = /vpc-routes
20172024-09-06T06:42:39.684ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57906
20182024-09-06T06:42:39.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57906 req_id = 52ac065a-7d75-4666-9b84-05f8d63b4b91 response_code = 200 uri = /v2p
20192024-09-06T06:42:39.921ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49479
20202024-09-06T06:42:39.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49479 req_id = 3e5f98b7-68ee-44fe-999d-93240df2dfa2 response_code = 200 uri = /v2p
20212024-09-06T06:42:40.759ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54511
20222024-09-06T06:42:40.760ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 684 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54511 req_id = 461c6954-3b74-4516-8490-f023e59ecc5d response_code = 200 uri = /vpc-routes
20232024-09-06T06:42:40.768ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54511 req_id = a760b721-9472-4e7a-b1f6-294e73804236 response_code = 204 uri = /vpc-routes