Max level shown:
1 [ Sep 24 04:29:37 Enabled. ]
2 [ Sep 24 04:29:37 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Sep 24 04:29:37 Method "start" exited with status 0. ]
4 [ Sep 24 04:29:37 Rereading configuration. ]
5 [ Sep 24 04:29:37 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-09-24T04:29:37.777ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-09-24T04:29:37.777ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-09-24T04:29:37.829ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-09-24T04:29:37.829ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-09-24T04:29:37.943ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-09-24T04:29:37.952ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-09-24T04:29:37.960ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-09-24T04:29:38.569ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-09-24T04:29:38.709ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-09-24T04:29:38.709ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-09-24T04:29:38.709ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-09-24T04:29:38.709ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-09-24T04:29:38.709ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-09-24T04:29:38.709ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-09-24T04:29:38.712ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-09-24T04:29:38.712ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-09-24T04:29:38.712ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-09-24T04:29:38.712ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-09-24T04:29:38.712ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-09-24T04:29:38.712ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-09-24T04:29:38.712ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-09-24T04:29:38.712ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-09-24T04:29:38.712ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-09-24T04:29:38.723ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-09-24T04:29:38.723ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool)
322024-09-24T04:29:38.822ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal }
332024-09-24T04:29:39.206ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-09-24T04:29:39.206ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-09-24T04:29:39.206ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-09-24T04:29:39.206ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-09-24T04:29:39.206ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-09-24T04:29:39.206ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-09-24T04:29:39.206ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-09-24T04:29:39.214ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-09-24T04:29:39.214ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 1933fe24-deab-4311-a300-a09ef207e36a (zpool)
422024-09-24T04:29:39.306ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal }
432024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
522024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
572024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
582024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
602024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
682024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
692024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
702024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
712024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
782024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
822024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
832024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
842024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-09-24T04:29:39.723ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-09-24T04:29:39.723ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-09-24T04:29:39.723ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-09-24T04:29:39.723ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-09-24T04:29:39.723ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
962024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
972024-09-24T04:29:39.723ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
982024-09-24T04:29:39.724ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-09-24T04:29:39.724ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1002024-09-24T04:29:39.724ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1012024-09-24T04:29:39.724ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1022024-09-24T04:29:39.724ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-09-24T04:29:39.724ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-09-24T04:29:39.724ZINFOSledAgent (bootstore): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-09-24T04:29:39.724ZINFOSledAgent (bootstore): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-09-24T04:29:39.724ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-09-24T04:29:39.724ZINFOSledAgent (bootstore): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-09-24T04:29:39.724ZINFOSledAgent (bootstore): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-09-24T04:29:39.724ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-09-24T04:29:39.724ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-09-24T04:29:39.724ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-09-24T04:29:39.734ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:737
1132024-09-24T04:29:39.734ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1371, tv_nsec: 418550696 } time_to_next_cleanup = 599.999925948s
1142024-09-24T04:29:39.734ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3917
1152024-09-24T04:29:39.734ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-09-24T04:29:39.734ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-09-24T04:29:39.734ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-09-24T04:29:39.737ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-09-24T04:29:39.739ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-09-24T04:29:39.739ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-09-24T04:29:39.739ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1222024-09-24T04:29:39.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-09-24T04:29:39.739ZINFOSledAgent (RSS): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-09-24T04:29:39.739ZINFOSledAgent (RSS): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-09-24T04:29:39.739ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-09-24T04:29:39.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-09-24T04:29:39.739ZINFOSledAgent (RSS): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-09-24T04:29:39.739ZINFOSledAgent (RSS): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-09-24T04:29:39.739ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-09-24T04:29:39.739ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1312024-09-24T04:29:39.739ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1322024-09-24T04:29:39.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-09-24T04:29:39.740ZINFOSledAgent (RSS): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-09-24T04:29:39.740ZINFOSledAgent (RSS): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-09-24T04:29:39.740ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 052f3529-69d0-4ca1-873b-88ee5f382b59, sleds: { [fdb0:18c0:4d0d:a02a::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 223823d0-6342-4be2-8483-3bbd7919ea67, rack_id: 052f3529-69d0-4ca1-873b-88ee5f382b59, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1362024-09-24T04:29:39.740ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-09-24T04:29:39.740ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-09-24T04:29:39.740ZINFOSledAgent (bootstore): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-09-24T04:29:39.740ZINFOSledAgent (bootstore): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-09-24T04:29:39.741ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-09-24T04:29:39.741ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 223823d0-6342-4be2-8483-3bbd7919ea67, rack_id: 052f3529-69d0-4ca1-873b-88ee5f382b59, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-09-24T04:29:39.742ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4214
1432024-09-24T04:29:39.742ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4596
1442024-09-24T04:29:39.746ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-09-24T04:29:39.751ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-09-24T04:29:39.751ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-09-24T04:29:39.756ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-09-24T04:29:39.756ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-09-24T04:29:39.756ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:a02a::1]:49189
1502024-09-24T04:29:39.756ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 223823d0-6342-4be2-8483-3bbd7919ea67, rack_id: 052f3529-69d0-4ca1-873b-88ee5f382b59, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:361
1512024-09-24T04:29:39.756ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:376
1522024-09-24T04:29:39.756ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1532024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1542024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1552024-09-24T04:29:39.756ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1562024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1572024-09-24T04:29:39.756ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1582024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-09-24T04:29:39.756ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-09-24T04:29:39.756ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-09-24T04:29:39.773ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-09-24T04:29:39.796ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-09-24T04:29:39.813ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1672024-09-24T04:29:39.836ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
1682024-09-24T04:29:39.836ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1692024-09-24T04:29:39.836ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
1702024-09-24T04:29:39.836ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1712024-09-24T04:29:39.839ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 87.848698ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5acb6576-7c3b-410a-b8df-f415b288131b", "content-length": "124", "date": "Tue, 24 Sep 2024 04:29:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5acb6576-7c3b-410a-b8df-f415b288131b" } file = clients/ddm-admin-client/src/lib.rs:177
1722024-09-24T04:29:39.849ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_1cd7b3c4-f332-469d-a6b7-9f7501e7549c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-09-24T04:29:39.872ZINFOSledAgent: successfully created encrypted zvol: name="oxi_1cd7b3c4-f332-469d-a6b7-9f7501e7549c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-09-24T04:29:39.872ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_1cd7b3c4-f332-469d-a6b7-9f7501e7549c/swap" file = sled-agent/src/swap_device.rs:96
1752024-09-24T04:29:39.878ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
1762024-09-24T04:29:39.878ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1772024-09-24T04:29:39.878ZINFOSledAgent (BackingFs): Ensuring dataset oxi_1cd7b3c4-f332-469d-a6b7-9f7501e7549c/backing/fmd file = sled-agent/src/backing_fs.rs:138
1782024-09-24T04:29:39.929ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-09-24T04:29:39.930ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 91.871386ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "69c54b4b-ed33-4d7e-a6d1-675fe9ca1586", "content-length": "124", "date": "Tue, 24 Sep 2024 04:29:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "69c54b4b-ed33-4d7e-a6d1-675fe9ca1586" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-09-24T04:29:39.963ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1812024-09-24T04:29:40.023ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-09-24T04:29:40.023ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 183.494302ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "334132f2-9538-49d4-9cd3-4496059c771d", "content-length": "124", "date": "Tue, 24 Sep 2024 04:29:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "334132f2-9538-49d4-9cd3-4496059c771d" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-09-24T04:29:40.182ZINFOSledAgent (BackingFs): Mounting oxi_1cd7b3c4-f332-469d-a6b7-9f7501e7549c/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1842024-09-24T04:29:40.195ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1852024-09-24T04:29:40.195ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1862024-09-24T04:29:40.195ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1872024-09-24T04:29:40.195ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1882024-09-24T04:29:40.208ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892024-09-24T04:29:40.208ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 176.718968ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "31b18583-c514-434a-8631-84d508b4c068", "content-length": "124", "date": "Tue, 24 Sep 2024 04:29:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "31b18583-c514-434a-8631-84d508b4c068" } file = clients/ddm-admin-client/src/lib.rs:177
1902024-09-24T04:29:40.385ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-09-24T04:29:40.386ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 139.165844ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2cbf3e79-3308-4675-b8c7-98dc1c21ebda", "content-length": "124", "date": "Tue, 24 Sep 2024 04:29:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2cbf3e79-3308-4675-b8c7-98dc1c21ebda" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-09-24T04:29:40.526ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-09-24T04:29:40.527ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 309.165619ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "33b62c87-0d1c-4862-924a-47e28cecbfe7", "content-length": "124", "date": "Tue, 24 Sep 2024 04:29:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "33b62c87-0d1c-4862-924a-47e28cecbfe7" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-09-24T04:29:40.836ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-09-24T04:29:40.837ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 277.139542ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "365be2ee-b71f-4391-b76f-7751b59f17a4", "content-length": "124", "date": "Tue, 24 Sep 2024 04:29:40 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "365be2ee-b71f-4391-b76f-7751b59f17a4" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-09-24T04:29:40.936ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
1972024-09-24T04:29:40.942ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:46023 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
1982024-09-24T04:29:40.942ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
1992024-09-24T04:29:40.942ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:46023 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 223823d0-6342-4be2-8483-3bbd7919ea67 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2002024-09-24T04:29:40.943ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 200.780776ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2012024-09-24T04:29:40.944ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2022024-09-24T04:29:40.944ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2032024-09-24T04:29:40.944ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:921 underlay_address = fd00:1122:3344:101::1
2042024-09-24T04:29:40.944ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
2052024-09-24T04:29:41.114ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2062024-09-24T04:29:41.146ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 330.888712ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2072024-09-24T04:29:41.478ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.240049798s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2082024-09-24T04:29:42.720ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.299116235s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2092024-09-24T04:29:43.438ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2670
2102024-09-24T04:29:43.439ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2888
2112024-09-24T04:29:43.439ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2540
2122024-09-24T04:29:43.439ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2587
2132024-09-24T04:29:43.439ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2969
2142024-09-24T04:29:43.439ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3011
2152024-09-24T04:29:43.440ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2813
2162024-09-24T04:29:43.442ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2581
2172024-09-24T04:29:43.442ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='a50dc56c-3bdf-4046-93f8-1e1c2a428b12'/> <propval type="astring" name="rack_id" value='052f3529-69d0-4ca1-873b-88ee5f382b59'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe25:95bb'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='223823d0-6342-4be2-8483-3bbd7919ea67'/> <propval type="astring" name="rack_id" value='052f3529-69d0-4ca1-873b-88ee5f382b59'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='223823d0-6342-4be2-8483-3bbd7919ea67'/> <propval type="astring" name="rack_uuid" value='052f3529-69d0-4ca1-873b-88ee5f382b59'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='223823d0-6342-4be2-8483-3bbd7919ea67'/> <propval type="astring" name="rack_uuid" value='052f3529-69d0-4ca1-873b-88ee5f382b59'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2182024-09-24T04:29:43.442ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2192024-09-24T04:29:43.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 55.350299ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-09-24T04:29:43.874ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 97.866751ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-09-24T04:29:44.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 163.892991ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-09-24T04:29:44.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 355.202403ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-09-24T04:29:44.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 735.505018ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-09-24T04:29:45.022ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.961015587s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2252024-09-24T04:29:45.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 728.688888ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-09-24T04:29:46.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.146680262s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-09-24T04:29:48.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 628.645561ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2282024-09-24T04:29:48.037ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.380800316s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2292024-09-24T04:29:48.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.386817449s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2302024-09-24T04:29:50.524ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2312024-09-24T04:29:50.524ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:431
2322024-09-24T04:29:50.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2332024-09-24T04:29:50.524ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2342024-09-24T04:29:50.524ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2352024-09-24T04:29:50.524ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2362024-09-24T04:29:50.524ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2372024-09-24T04:29:50.524ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3917
2382024-09-24T04:29:50.524ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:a02a::1]:49189
2392024-09-24T04:29:50.758ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2402024-09-24T04:29:50.758ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2412024-09-24T04:29:50.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2422024-09-24T04:29:50.758ZINFOSledAgent (RSS): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2432024-09-24T04:29:50.758ZINFOSledAgent (RSS): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2442024-09-24T04:29:50.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-09-24T04:29:50.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-09-24T04:29:50.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2472024-09-24T04:29:50.763ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4238 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2482024-09-24T04:29:50.763ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4254
2492024-09-24T04:29:50.763ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2502024-09-24T04:29:50.992ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2512024-09-24T04:29:50.993ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2522024-09-24T04:29:51.019ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55592
2532024-09-24T04:29:51.020ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2542024-09-24T04:29:51.020ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 328 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55592 req_id = 7017ced5-cf95-48bf-956b-4f2ad7a911a6 response_code = 200 uri = /inventory
2552024-09-24T04:29:51.050ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39885
2562024-09-24T04:29:51.050ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39885 req_id = 1e059312-df1f-4a53-8570-1225f0f4939d response_code = 200 uri = /sled-role
2572024-09-24T04:29:51.051ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2582024-09-24T04:29:51.051ZINFOSledAgent (RSS): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2592024-09-24T04:29:51.052ZINFOSledAgent (RSS): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2602024-09-24T04:29:51.052ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:1001
2612024-09-24T04:29:51.083ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: f352320b-33cd-4094-9d78-b70adfcd57ce, pool_id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: dedcd2b5-0d1a-42d1-8c1e-244c1dd1b23b, pool_id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 9d747df9-fa8e-4c6c-bbac-7cbf74bb3173, pool_id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: a9bd8ffb-a39d-4acf-ac4a-9a2172a274fc, pool_id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 30dbb8c5-c5cf-4e2d-9f4a-56e1fc10bc8f, pool_id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 46275c71-672a-4e8e-8a68-bebecd0b35c3, pool_id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 58cc1db4-1902-4c20-9c24-ab19592b9c6e, pool_id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 39696cd3-d565-4e65-9b8d-baf1f97e97c1, pool_id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: aff60c0f-ca71-4994-b9c5-5687bef594b1, pool_id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2622024-09-24T04:29:51.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46616
2632024-09-24T04:29:51.083ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:864 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2642024-09-24T04:29:51.083ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: f352320b-33cd-4094-9d78-b70adfcd57ce, pool_id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: dedcd2b5-0d1a-42d1-8c1e-244c1dd1b23b, pool_id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 9d747df9-fa8e-4c6c-bbac-7cbf74bb3173, pool_id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: a9bd8ffb-a39d-4acf-ac4a-9a2172a274fc, pool_id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 30dbb8c5-c5cf-4e2d-9f4a-56e1fc10bc8f, pool_id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 46275c71-672a-4e8e-8a68-bebecd0b35c3, pool_id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 58cc1db4-1902-4c20-9c24-ab19592b9c6e, pool_id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 39696cd3-d565-4e65-9b8d-baf1f97e97c1, pool_id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: aff60c0f-ca71-4994-b9c5-5687bef594b1, pool_id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2652024-09-24T04:29:51.083ZINFOSledAgent (StorageManager): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2662024-09-24T04:29:51.083ZINFOSledAgent (StorageManager): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2672024-09-24T04:29:51.083ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2682024-09-24T04:29:51.083ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2692024-09-24T04:29:51.083ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2702024-09-24T04:29:51.083ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2712024-09-24T04:29:51.083ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2722024-09-24T04:29:51.224ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2732024-09-24T04:29:51.233ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2742024-09-24T04:29:51.233ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool)
2752024-09-24T04:29:51.328ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }
2762024-09-24T04:29:51.335ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2772024-09-24T04:29:51.335ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2782024-09-24T04:29:51.336ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2792024-09-24T04:29:51.336ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2802024-09-24T04:29:51.336ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2812024-09-24T04:29:51.336ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2822024-09-24T04:29:51.336ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2832024-09-24T04:29:51.371ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2842024-09-24T04:29:51.379ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_036ed427-877d-4d45-ad05-e0fad8a2b1c2/crypt/zone file = sled-storage/src/dataset.rs:280
2852024-09-24T04:29:51.527ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
2862024-09-24T04:29:51.549ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_036ed427-877d-4d45-ad05-e0fad8a2b1c2 file = sled-storage/src/dataset.rs:366
2872024-09-24T04:29:51.556ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2882024-09-24T04:29:51.556ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2892024-09-24T04:29:51.556ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2902024-09-24T04:29:51.556ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2912024-09-24T04:29:51.556ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2922024-09-24T04:29:51.566ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2932024-09-24T04:29:51.566ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool)
2942024-09-24T04:29:51.659ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }
2952024-09-24T04:29:51.667ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2962024-09-24T04:29:51.667ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
2972024-09-24T04:29:51.667ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
2982024-09-24T04:29:51.667ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
2992024-09-24T04:29:51.667ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3002024-09-24T04:29:51.667ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3012024-09-24T04:29:51.667ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3022024-09-24T04:29:51.703ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3032024-09-24T04:29:51.711ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_0f5ed56b-52e5-409d-bf51-72e9cba37d51/crypt/zone file = sled-storage/src/dataset.rs:280
3042024-09-24T04:29:51.879ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_0f5ed56b-52e5-409d-bf51-72e9cba37d51 file = sled-storage/src/dataset.rs:366
3052024-09-24T04:29:51.886ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3062024-09-24T04:29:51.886ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3072024-09-24T04:29:51.887ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3082024-09-24T04:29:51.887ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3092024-09-24T04:29:51.887ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3102024-09-24T04:29:51.896ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3112024-09-24T04:29:51.896ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool)
3122024-09-24T04:29:51.986ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }
3132024-09-24T04:29:51.994ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3142024-09-24T04:29:51.994ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3152024-09-24T04:29:51.994ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3162024-09-24T04:29:51.994ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3172024-09-24T04:29:51.994ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3182024-09-24T04:29:51.994ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3192024-09-24T04:29:51.994ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3202024-09-24T04:29:52.030ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3212024-09-24T04:29:52.038ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef/crypt/zone file = sled-storage/src/dataset.rs:280
3222024-09-24T04:29:52.214ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef file = sled-storage/src/dataset.rs:366
3232024-09-24T04:29:52.223ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3242024-09-24T04:29:52.223ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3252024-09-24T04:29:52.223ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3262024-09-24T04:29:52.223ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3272024-09-24T04:29:52.223ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3282024-09-24T04:29:52.233ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3292024-09-24T04:29:52.233ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool)
3302024-09-24T04:29:52.327ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }
3312024-09-24T04:29:52.335ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3322024-09-24T04:29:52.335ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3332024-09-24T04:29:52.335ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3342024-09-24T04:29:52.335ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3352024-09-24T04:29:52.336ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3362024-09-24T04:29:52.336ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3372024-09-24T04:29:52.336ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3382024-09-24T04:29:52.371ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3392024-09-24T04:29:52.379ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7a7d72d1-c18a-4053-8666-72377ed40c63/crypt/zone file = sled-storage/src/dataset.rs:280
3402024-09-24T04:29:52.530ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
3412024-09-24T04:29:52.539ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7a7d72d1-c18a-4053-8666-72377ed40c63 file = sled-storage/src/dataset.rs:366
3422024-09-24T04:29:52.546ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3432024-09-24T04:29:52.546ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3442024-09-24T04:29:52.546ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3452024-09-24T04:29:52.546ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3462024-09-24T04:29:52.546ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3472024-09-24T04:29:52.556ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3482024-09-24T04:29:52.556ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool)
3492024-09-24T04:29:52.651ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }
3502024-09-24T04:29:52.658ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3512024-09-24T04:29:52.659ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3522024-09-24T04:29:52.659ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3532024-09-24T04:29:52.659ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3542024-09-24T04:29:52.659ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3552024-09-24T04:29:52.659ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3562024-09-24T04:29:52.659ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3572024-09-24T04:29:52.695ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3582024-09-24T04:29:52.703ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cf2b5393-7938-4187-81ff-eefd54b8dca3/crypt/zone file = sled-storage/src/dataset.rs:280
3592024-09-24T04:29:52.875ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cf2b5393-7938-4187-81ff-eefd54b8dca3 file = sled-storage/src/dataset.rs:366
3602024-09-24T04:29:52.882ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3612024-09-24T04:29:52.882ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3622024-09-24T04:29:52.882ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3632024-09-24T04:29:52.882ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3642024-09-24T04:29:52.882ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3652024-09-24T04:29:52.892ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3662024-09-24T04:29:52.892ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool)
3672024-09-24T04:29:52.988ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool), kind: External }
3682024-09-24T04:29:52.996ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3692024-09-24T04:29:52.996ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3702024-09-24T04:29:52.996ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3712024-09-24T04:29:52.996ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3722024-09-24T04:29:52.996ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3732024-09-24T04:29:52.996ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3742024-09-24T04:29:52.996ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3752024-09-24T04:29:53.032ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3762024-09-24T04:29:53.040ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_aa8dd6da-a2be-4c64-ab22-604023652cfd/crypt/zone file = sled-storage/src/dataset.rs:280
3772024-09-24T04:29:53.219ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_aa8dd6da-a2be-4c64-ab22-604023652cfd file = sled-storage/src/dataset.rs:366
3782024-09-24T04:29:53.228ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3792024-09-24T04:29:53.228ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3802024-09-24T04:29:53.228ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3812024-09-24T04:29:53.228ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3822024-09-24T04:29:53.228ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3832024-09-24T04:29:53.239ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3842024-09-24T04:29:53.239ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = a09f232d-c31e-4961-bb00-74c2dac76055 (zpool)
3852024-09-24T04:29:53.335ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External }
3862024-09-24T04:29:53.343ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3872024-09-24T04:29:53.343ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3882024-09-24T04:29:53.343ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3892024-09-24T04:29:53.343ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3902024-09-24T04:29:53.343ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3912024-09-24T04:29:53.344ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3922024-09-24T04:29:53.344ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3932024-09-24T04:29:53.381ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3942024-09-24T04:29:53.390ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a09f232d-c31e-4961-bb00-74c2dac76055/crypt/zone file = sled-storage/src/dataset.rs:280
3952024-09-24T04:29:53.533ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
3962024-09-24T04:29:53.570ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a09f232d-c31e-4961-bb00-74c2dac76055 file = sled-storage/src/dataset.rs:366
3972024-09-24T04:29:53.578ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3982024-09-24T04:29:53.578ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3992024-09-24T04:29:53.578ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4002024-09-24T04:29:53.578ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4012024-09-24T04:29:53.579ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4022024-09-24T04:29:53.589ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
4032024-09-24T04:29:53.589ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = f46073e5-0cf7-4937-b820-b3dac0544ced (zpool)
4042024-09-24T04:29:53.687ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External }
4052024-09-24T04:29:53.696ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4062024-09-24T04:29:53.696ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4072024-09-24T04:29:53.696ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4082024-09-24T04:29:53.696ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4092024-09-24T04:29:53.696ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4102024-09-24T04:29:53.696ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4112024-09-24T04:29:53.696ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4122024-09-24T04:29:53.733ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4132024-09-24T04:29:53.741ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f46073e5-0cf7-4937-b820-b3dac0544ced/crypt/zone file = sled-storage/src/dataset.rs:280
4142024-09-24T04:29:53.930ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f46073e5-0cf7-4937-b820-b3dac0544ced file = sled-storage/src/dataset.rs:366
4152024-09-24T04:29:53.938ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4162024-09-24T04:29:53.938ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4172024-09-24T04:29:53.938ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4182024-09-24T04:29:53.938ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4192024-09-24T04:29:53.938ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4202024-09-24T04:29:53.949ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4212024-09-24T04:29:53.949ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool)
4222024-09-24T04:29:54.043ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool), kind: External }
4232024-09-24T04:29:54.052ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4242024-09-24T04:29:54.052ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4252024-09-24T04:29:54.052ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4262024-09-24T04:29:54.052ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4272024-09-24T04:29:54.052ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4282024-09-24T04:29:54.052ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4292024-09-24T04:29:54.052ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4302024-09-24T04:29:54.086ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4312024-09-24T04:29:54.095ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_5bcd6732-e8a3-402d-925d-d2e58792effe/crypt/zone file = sled-storage/src/dataset.rs:280
4322024-09-24T04:29:54.272ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_5bcd6732-e8a3-402d-925d-d2e58792effe file = sled-storage/src/dataset.rs:366
4332024-09-24T04:29:54.281ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4342024-09-24T04:29:54.281ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4352024-09-24T04:29:54.281ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4362024-09-24T04:29:54.281ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1cd7b3c4-f332-469d-a6b7-9f7501e7549c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1933fe24-deab-4311-a300-a09ef207e36a (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4372024-09-24T04:29:54.281ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4382024-09-24T04:29:54.281ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4392024-09-24T04:29:54.281ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4402024-09-24T04:29:54.281ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:868 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4412024-09-24T04:29:54.281ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4422024-09-24T04:29:54.281ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:894 generation = Generation(1) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4432024-09-24T04:29:54.281ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:899 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4442024-09-24T04:29:54.281ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:904 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4452024-09-24T04:29:54.281ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4462024-09-24T04:29:54.281ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:910 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4472024-09-24T04:29:54.281ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:773 gen = Generation(1)
4482024-09-24T04:29:54.281ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:915 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4492024-09-24T04:29:54.281ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3198267 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46616 req_id = decdfd82-1816-44cb-bafe-a4c7d7bf043f response_code = 200 uri = /omicron-physical-disks
4502024-09-24T04:29:54.312ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 6b582b65-02df-47ca-87c9-8010b32bb48c (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2756a2fa-a955-4f2e-9728-0994c38722e2 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 6d05b27d-c7f2-4261-964b-90b330b71455 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4512024-09-24T04:29:54.313ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38031
4522024-09-24T04:29:54.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4532024-09-24T04:29:54.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4542024-09-24T04:29:54.363ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4552024-09-24T04:29:54.363ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4562024-09-24T04:29:54.412ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4572024-09-24T04:29:54.412ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }
4582024-09-24T04:29:54.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4592024-09-24T04:29:54.461ZINFOSledAgent (ServiceManager): No ledger in /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-zones.json file = common/src/ledger.rs:188
4602024-09-24T04:29:54.461ZINFOSledAgent (ServiceManager): No ledger in /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-zones.json file = common/src/ledger.rs:188
4612024-09-24T04:29:54.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4622024-09-24T04:29:54.503ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4632024-09-24T04:29:54.503ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4642024-09-24T04:29:54.503ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4652024-09-24T04:29:54.538ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4662024-09-24T04:29:54.556ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455 file = illumos-utils/src/zone.rs:316
4672024-09-24T04:29:54.560ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c file = illumos-utils/src/zone.rs:316
4682024-09-24T04:29:54.564ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2 file = illumos-utils/src/zone.rs:316
4692024-09-24T04:29:54.576ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455 file = illumos-utils/src/zone.rs:354
4702024-09-24T04:29:54.580ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c file = illumos-utils/src/zone.rs:354
4712024-09-24T04:29:54.584ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2 file = illumos-utils/src/zone.rs:354
4722024-09-24T04:29:55.540ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
4732024-09-24T04:29:56.071ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4742024-09-24T04:29:56.071ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4752024-09-24T04:29:56.071ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
4762024-09-24T04:29:56.556ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4772024-09-24T04:29:56.556ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4782024-09-24T04:29:57.036ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4792024-09-24T04:29:57.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 36.995497ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
4802024-09-24T04:29:57.043ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
4812024-09-24T04:29:57.047ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4822024-09-24T04:29:57.047ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
4832024-09-24T04:29:57.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 129.097664ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
4842024-09-24T04:29:57.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 25.065197ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
4852024-09-24T04:29:57.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 38.745519ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
4862024-09-24T04:29:57.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 135.229429ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
4872024-09-24T04:29:57.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 290.293373ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
4882024-09-24T04:29:57.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 80.972373ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
4892024-09-24T04:29:57.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 105.49915ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
4902024-09-24T04:29:57.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 164.290818ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
4912024-09-24T04:29:57.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 576.082996ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
4922024-09-24T04:29:57.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 393.089166ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
4932024-09-24T04:29:57.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 374.312109ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
4942024-09-24T04:29:57.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 894.053717ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
4952024-09-24T04:29:58.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 838.822933ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
4962024-09-24T04:29:58.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 427.813238ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
4972024-09-24T04:29:58.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 540.730557ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
4982024-09-24T04:29:58.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 889.661558ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
4992024-09-24T04:29:58.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 1.09670057s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
5002024-09-24T04:29:59.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 523.316401ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
5012024-09-24T04:29:59.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 1.065813272s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
5022024-09-24T04:29:59.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 950.13267ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
5032024-09-24T04:29:59.911ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4262
5042024-09-24T04:29:59.911ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4272 gateway = fd00:1122:3344:101::1
5052024-09-24T04:29:59.918ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4351
5062024-09-24T04:30:00.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 848.193283ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
5072024-09-24T04:30:00.041ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4399
5082024-09-24T04:30:00.041ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4426
5092024-09-24T04:30:00.077ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4439
5102024-09-24T04:30:00.077ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4303
5112024-09-24T04:30:00.127ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4345
5122024-09-24T04:30:00.127ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4406 rack_subnet = fd00:1122:3344::
5132024-09-24T04:30:00.148ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4417
5142024-09-24T04:30:00.148ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4459
5152024-09-24T04:30:00.243ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4491
5162024-09-24T04:30:00.243ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4497
5172024-09-24T04:30:00.354ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4538
5182024-09-24T04:30:00.355ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5192024-09-24T04:30:00.387ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:571 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
5202024-09-24T04:30:00.387ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 42.465896ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5212024-09-24T04:30:00.431ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 100.184371ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5222024-09-24T04:30:00.526ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9121874546550192045 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
5232024-09-24T04:30:00.526ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16136884810500391095 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
5242024-09-24T04:30:00.533ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 225.041595ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5252024-09-24T04:30:00.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2") failed: Property not found. retry in 1.411426675s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
5262024-09-24T04:30:00.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455") failed: Property not found. retry in 1.488207573s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
5272024-09-24T04:30:00.759ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 428.881221ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5282024-09-24T04:30:00.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c") failed: Property not found. retry in 1.109733862s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
5292024-09-24T04:30:00.897ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:43397
5302024-09-24T04:30:00.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 620 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:43397 req_id = 6b84a3c3-78a5-4ef4-9ee1-349a08ed20f7 response_code = 200 uri = /sled-identifiers
5312024-09-24T04:30:00.995ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
5322024-09-24T04:30:00.995ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
5332024-09-24T04:30:01.190ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5342024-09-24T04:30:01.222ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5352024-09-24T04:30:01.222ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5362024-09-24T04:30:01.223ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 1
5372024-09-24T04:30:01.223ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5382024-09-24T04:30:01.260ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
5392024-09-24T04:30:01.310ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5402024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5412024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5422024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5432024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5442024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5452024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5462024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5472024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5482024-09-24T04:30:01.310ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5492024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5502024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5512024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5522024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5532024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5542024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5552024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5562024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5572024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5582024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5592024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5602024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5612024-09-24T04:30:01.311ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5622024-09-24T04:30:02.272ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_6d05b27d-c7f2-4261-964b-90b330b71455
5632024-09-24T04:30:02.272ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_6b582b65-02df-47ca-87c9-8010b32bb48c
5642024-09-24T04:30:02.272ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_2756a2fa-a955-4f2e-9728-0994c38722e2
5652024-09-24T04:30:02.272ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5662024-09-24T04:30:02.273ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5672024-09-24T04:30:02.273ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7960299 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38031 req_id = ce3c989e-c3f5-49ba-b753-156869056a36 response_code = 204 uri = /omicron-zones
5682024-09-24T04:30:02.275ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5692024-09-24T04:30:02.305ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 110.362969ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:595
5702024-09-24T04:30:02.442ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5712024-09-24T04:30:02.554ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 134.626156ms) dns_config_addr = [fd00:1122:3344:2::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:2::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:595
5722024-09-24T04:30:02.707ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5732024-09-24T04:30:02.754ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5742024-09-24T04:30:02.754ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5752024-09-24T04:30:02.787ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5762024-09-24T04:30:02.787ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5772024-09-24T04:30:02.787ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5782024-09-24T04:30:02.816ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 6b582b65-02df-47ca-87c9-8010b32bb48c (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2756a2fa-a955-4f2e-9728-0994c38722e2 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 6d05b27d-c7f2-4261-964b-90b330b71455 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 14c78b5e-89d0-46da-a2db-dc5f845a900e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: c657bb17-8479-480d-b836-9c6e659d7897, kind: Service { id: 14c78b5e-89d0-46da-a2db-dc5f845a900e }, name: Name("ntp-14c78b5e-89d0-46da-a2db-dc5f845a900e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 184, 8])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5792024-09-24T04:30:02.817ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34279
5802024-09-24T04:30:02.817ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5812024-09-24T04:30:02.817ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5822024-09-24T04:30:02.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5832024-09-24T04:30:02.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5842024-09-24T04:30:02.858ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5852024-09-24T04:30:02.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }
5862024-09-24T04:30:02.878ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5872024-09-24T04:30:02.878ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-zones.json file = common/src/ledger.rs:177
5882024-09-24T04:30:02.878ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-zones.json file = common/src/ledger.rs:177
5892024-09-24T04:30:02.878ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5902024-09-24T04:30:02.890ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5912024-09-24T04:30:02.922ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5922024-09-24T04:30:02.923ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5932024-09-24T04:30:02.923ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5942024-09-24T04:30:02.953ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 184, 8]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5952024-09-24T04:30:02.953ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = boundary_ntp
5962024-09-24T04:30:02.953ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
5972024-09-24T04:30:02.953ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
5982024-09-24T04:30:02.954ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
5992024-09-24T04:30:02.954ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6002024-09-24T04:30:02.982ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e file = illumos-utils/src/zone.rs:316
6012024-09-24T04:30:03.000ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e file = illumos-utils/src/zone.rs:354
6022024-09-24T04:30:03.631ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6032024-09-24T04:30:03.631ZINFOSledAgent (ServiceManager): Booting oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6042024-09-24T04:30:03.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 51.863734ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6052024-09-24T04:30:03.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 144.11821ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6062024-09-24T04:30:03.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 129.835055ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6072024-09-24T04:30:04.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 331.26052ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6082024-09-24T04:30:04.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 794.094996ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6092024-09-24T04:30:05.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 1.001788929s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6102024-09-24T04:30:06.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 661.159156ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6112024-09-24T04:30:06.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 1.462129568s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6122024-09-24T04:30:08.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e") failed: Property not found. retry in 983.48174ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6132024-09-24T04:30:09.443ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e
6142024-09-24T04:30:09.443ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6152024-09-24T04:30:09.443ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6162024-09-24T04:30:09.444ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6627131 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34279 req_id = 8c6a9d8a-026e-449d-a8d7-a2063622458f response_code = 204 uri = /omicron-zones
6172024-09-24T04:30:09.446ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6182024-09-24T04:30:09.477ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6192024-09-24T04:30:09.477ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54995
6202024-09-24T04:30:09.485ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6212024-09-24T04:30:09.485ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8251 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54995 req_id = 1b6fb3e5-34aa-480e-8862-9b4f41ce0d4d response_code = 500 uri = /timesync
6222024-09-24T04:30:09.488ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6232024-09-24T04:30:09.662ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6242024-09-24T04:30:09.662ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47647
6252024-09-24T04:30:10.528ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9121874546550192045 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6262024-09-24T04:30:10.528ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16136884810500391095 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6272024-09-24T04:30:10.996ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6282024-09-24T04:30:10.996ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6292024-09-24T04:30:11.995ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12313287160412236835 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6302024-09-24T04:30:12.165ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17872697599083210197 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6312024-09-24T04:30:12.275ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2675747394570014431 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6322024-09-24T04:30:13.670ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6332024-09-24T04:30:13.670ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4008132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47647 req_id = 1eaa962d-ba81-4916-b483-0764e35a0718 response_code = 500 uri = /timesync
6342024-09-24T04:30:13.673ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6352024-09-24T04:30:13.786ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6362024-09-24T04:30:13.787ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50376
6372024-09-24T04:30:13.796ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6382024-09-24T04:30:13.796ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 9268 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50376 req_id = 0fbc8780-6df3-40c8-bbf0-c3cbd966533a response_code = 500 uri = /timesync
6392024-09-24T04:30:13.799ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6402024-09-24T04:30:13.907ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6412024-09-24T04:30:13.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43614
6422024-09-24T04:30:17.733ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_14c78b5e-89d0-46da-a2db-dc5f845a900e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6432024-09-24T04:30:17.733ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4009362 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43614 req_id = 773c6d12-cd66-4461-9f3d-c951b35e9bf6 response_code = 500 uri = /timesync
6442024-09-24T04:30:17.736ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6452024-09-24T04:30:17.894ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6462024-09-24T04:30:17.894ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37956
6472024-09-24T04:30:17.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9726 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37956 req_id = b17dc997-2619-4d51-9781-51ef7f187443 response_code = 200 uri = /timesync
6482024-09-24T04:30:17.908ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.284742296 } file = sled-agent/src/rack_setup/service.rs:660
6492024-09-24T04:30:17.908ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6502024-09-24T04:30:18.104ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6512024-09-24T04:30:18.104ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56755
6522024-09-24T04:30:18.113ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8817 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56755 req_id = 5dabf622-4b97-410e-81f8-2255e34cd836 response_code = 200 uri = /timesync
6532024-09-24T04:30:18.117ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.284175545 } file = sled-agent/src/rack_setup/service.rs:660
6542024-09-24T04:30:18.117ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6552024-09-24T04:30:18.299ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6562024-09-24T04:30:18.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61891
6572024-09-24T04:30:18.306ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 291.987012ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6582024-09-24T04:30:18.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8536 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61891 req_id = 0b356b12-f068-439c-b437-52bd5d4980f7 response_code = 200 uri = /timesync
6592024-09-24T04:30:18.310ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.283649325 } file = sled-agent/src/rack_setup/service.rs:660
6602024-09-24T04:30:18.310ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6612024-09-24T04:30:18.719ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6622024-09-24T04:30:18.719ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56079
6632024-09-24T04:30:18.728ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8609 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56079 req_id = 881bfb1b-21af-4840-85c8-abe7b6735fb1 response_code = 200 uri = /timesync
6642024-09-24T04:30:18.730ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152218.919143, correction: 0.284333199 } file = sled-agent/src/rack_setup/service.rs:660
6652024-09-24T04:30:18.730ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6662024-09-24T04:30:19.265ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14962262056481753924 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6672024-09-24T04:30:19.265ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3091031820373381064 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6682024-09-24T04:30:19.296ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6692024-09-24T04:30:19.296ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62438
6702024-09-24T04:30:19.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8496 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62438 req_id = bcd8cdc4-3559-470f-905c-a4594d40a70b response_code = 200 uri = /timesync
6712024-09-24T04:30:19.307ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1727152218.230102, correction: 0.282777071 } file = sled-agent/src/rack_setup/service.rs:660
6722024-09-24T04:30:19.307ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6732024-09-24T04:30:19.606ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6742024-09-24T04:30:19.607ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36775
6752024-09-24T04:30:19.615ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8655 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36775 req_id = 65aba012-3d79-4529-a454-53edad17a3dc response_code = 200 uri = /timesync
6762024-09-24T04:30:19.617ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1727152218.230102, correction: 0.28193745 } file = sled-agent/src/rack_setup/service.rs:660
6772024-09-24T04:30:19.617ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6782024-09-24T04:30:19.919ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6792024-09-24T04:30:19.919ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63447
6802024-09-24T04:30:19.927ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8442 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63447 req_id = 50a2653d-3677-45c3-b2ea-7592e849589b response_code = 200 uri = /timesync
6812024-09-24T04:30:19.929ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1727152218.230102, correction: 0.281887472 } file = sled-agent/src/rack_setup/service.rs:660
6822024-09-24T04:30:19.929ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6832024-09-24T04:30:20.354ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9121874546550192045 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6842024-09-24T04:30:20.354ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16136884810500391095 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6852024-09-24T04:30:20.525ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6862024-09-24T04:30:20.525ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64940
6872024-09-24T04:30:20.534ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8723 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64940 req_id = 4d623504-c1f2-4cbf-aef3-a26993e52a52 response_code = 200 uri = /timesync
6882024-09-24T04:30:20.536ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1727152218.230102, correction: 0.280249923 } file = sled-agent/src/rack_setup/service.rs:660
6892024-09-24T04:30:20.536ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6902024-09-24T04:30:20.821ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6912024-09-24T04:30:20.821ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6922024-09-24T04:30:21.410ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6932024-09-24T04:30:21.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50566
6942024-09-24T04:30:21.420ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9437 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50566 req_id = e212c0c1-876c-40c6-95b2-564ecf6fe376 response_code = 200 uri = /timesync
6952024-09-24T04:30:21.422ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1727152218.230102, correction: 0.27880913 } file = sled-agent/src/rack_setup/service.rs:660
6962024-09-24T04:30:21.422ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6972024-09-24T04:30:21.824ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12313287160412236835 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
6982024-09-24T04:30:22.001ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6992024-09-24T04:30:22.001ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42506
7002024-09-24T04:30:22.001ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17872697599083210197 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7012024-09-24T04:30:22.011ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9522 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42506 req_id = 4e0c3c43-1638-4f67-a842-7127a4041f41 response_code = 200 uri = /timesync
7022024-09-24T04:30:22.013ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152221.0880609, correction: 0.27784124 } file = sled-agent/src/rack_setup/service.rs:660
7032024-09-24T04:30:22.013ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7042024-09-24T04:30:22.105ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2675747394570014431 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7052024-09-24T04:30:23.050ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7062024-09-24T04:30:23.051ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60224
7072024-09-24T04:30:23.059ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8646 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60224 req_id = e19e0c1a-88fe-42ae-89af-6c8ad47ecec1 response_code = 200 uri = /timesync
7082024-09-24T04:30:23.062ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152222.6692033, correction: 0.274972051 } file = sled-agent/src/rack_setup/service.rs:660
7092024-09-24T04:30:23.062ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7102024-09-24T04:30:24.842ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7112024-09-24T04:30:24.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54317
7122024-09-24T04:30:24.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8627 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54317 req_id = 33db7e85-ea75-466b-b95d-85fafd71c075 response_code = 200 uri = /timesync
7132024-09-24T04:30:24.855ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152224.945166, correction: 0.269963592 } file = sled-agent/src/rack_setup/service.rs:660
7142024-09-24T04:30:24.855ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7152024-09-24T04:30:26.069ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7162024-09-24T04:30:26.070ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33816
7172024-09-24T04:30:26.080ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9942 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33816 req_id = ba572e2b-1518-482c-9889-4ff3e7a3b676 response_code = 200 uri = /timesync
7182024-09-24T04:30:26.081ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152225.4898689, correction: 0.266629398 } file = sled-agent/src/rack_setup/service.rs:660
7192024-09-24T04:30:26.081ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7202024-09-24T04:30:27.656ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7212024-09-24T04:30:27.656ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35066
7222024-09-24T04:30:27.666ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9589 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35066 req_id = 5c630412-d9c6-4db1-93fa-f2a916252f02 response_code = 200 uri = /timesync
7232024-09-24T04:30:27.668ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152226.4525814, correction: 0.262290299 } file = sled-agent/src/rack_setup/service.rs:660
7242024-09-24T04:30:27.668ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7252024-09-24T04:30:29.295ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14962262056481753924 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7262024-09-24T04:30:29.295ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3091031820373381064 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7272024-09-24T04:30:29.970ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47213
7282024-09-24T04:30:29.970ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47213 req_id = 33babd3c-adbf-4e73-bb7c-5e490e6cd909 response_code = 200 uri = /baseboard
7292024-09-24T04:30:30.203ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7302024-09-24T04:30:30.203ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43954
7312024-09-24T04:30:30.213ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9223 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43954 req_id = ce58cece-3f28-4d0c-af07-1a2f0ed5ed30 response_code = 200 uri = /timesync
7322024-09-24T04:30:30.215ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152230.0513828, correction: 0.255425006 } file = sled-agent/src/rack_setup/service.rs:660
7332024-09-24T04:30:30.215ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7342024-09-24T04:30:30.384ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9121874546550192045 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7352024-09-24T04:30:30.384ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16136884810500391095 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7362024-09-24T04:30:30.850ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7372024-09-24T04:30:30.850ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7382024-09-24T04:30:31.853ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12313287160412236835 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7392024-09-24T04:30:32.029ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17872697599083210197 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7402024-09-24T04:30:32.133ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2675747394570014431 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7412024-09-24T04:30:32.261ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7422024-09-24T04:30:32.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55576
7432024-09-24T04:30:32.270ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8620 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55576 req_id = 6a969284-a925-4546-b53e-12cf2e3d42a4 response_code = 200 uri = /timesync
7442024-09-24T04:30:32.272ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152232.1053274, correction: 0.249880522 } file = sled-agent/src/rack_setup/service.rs:660
7452024-09-24T04:30:32.272ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7462024-09-24T04:30:33.675ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 435.076954ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7472024-09-24T04:30:34.519ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7482024-09-24T04:30:34.519ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40361
7492024-09-24T04:30:34.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9892 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40361 req_id = debf793d-64c6-4d55-8334-ac886faead47 response_code = 200 uri = /timesync
7502024-09-24T04:30:34.531ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152232.2111175, correction: 0.243781239 } file = sled-agent/src/rack_setup/service.rs:660
7512024-09-24T04:30:34.531ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7522024-09-24T04:30:37.844ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7532024-09-24T04:30:37.844ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48430
7542024-09-24T04:30:37.854ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48430 req_id = 3c024617-f8ee-44c1-877a-09798938392a response_code = 200 uri = /timesync
7552024-09-24T04:30:37.855ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152237.6405597, correction: 0.234791309 } file = sled-agent/src/rack_setup/service.rs:660
7562024-09-24T04:30:37.855ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7572024-09-24T04:30:39.325ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14962262056481753924 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7582024-09-24T04:30:39.325ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3091031820373381064 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7592024-09-24T04:30:40.414ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9121874546550192045 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7602024-09-24T04:30:40.414ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16136884810500391095 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7612024-09-24T04:30:40.880ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7622024-09-24T04:30:40.880ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7632024-09-24T04:30:41.706ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7642024-09-24T04:30:41.706ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44028
7652024-09-24T04:30:41.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8975 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44028 req_id = edf19d69-bacd-4808-8673-08a1c3b0f91c response_code = 200 uri = /timesync
7662024-09-24T04:30:41.718ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152240.9139152, correction: 0.224371538 } file = sled-agent/src/rack_setup/service.rs:660
7672024-09-24T04:30:41.718ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7682024-09-24T04:30:41.883ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12313287160412236835 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7692024-09-24T04:30:42.057ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17872697599083210197 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7702024-09-24T04:30:42.161ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2675747394570014431 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7712024-09-24T04:30:46.689ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7722024-09-24T04:30:46.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38937
7732024-09-24T04:30:46.699ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38937 req_id = d6168b73-3d74-4aaf-9ee8-66772c013f43 response_code = 200 uri = /timesync
7742024-09-24T04:30:46.701ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152244.9705186, correction: 0.210884362 } file = sled-agent/src/rack_setup/service.rs:660
7752024-09-24T04:30:46.701ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7762024-09-24T04:30:49.187ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 558.185531ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7772024-09-24T04:30:49.354ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14962262056481753924 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7782024-09-24T04:30:49.354ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3091031820373381064 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7792024-09-24T04:30:50.442ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9121874546550192045 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7802024-09-24T04:30:50.443ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16136884810500391095 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7812024-09-24T04:30:50.909ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7822024-09-24T04:30:50.909ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7832024-09-24T04:30:51.912ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12313287160412236835 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7842024-09-24T04:30:52.085ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17872697599083210197 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7852024-09-24T04:30:52.189ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2675747394570014431 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7862024-09-24T04:30:56.285ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7872024-09-24T04:30:56.286ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33429
7882024-09-24T04:30:56.295ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33429 req_id = bb880eb4-d7be-4a90-b92d-bf895a93a079 response_code = 200 uri = /timesync
7892024-09-24T04:30:56.298ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152255.608368, correction: 0.184935138 } file = sled-agent/src/rack_setup/service.rs:660
7902024-09-24T04:30:56.298ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7912024-09-24T04:30:59.383ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14962262056481753924 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7922024-09-24T04:30:59.383ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3091031820373381064 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7932024-09-24T04:31:00.086ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:56494
7942024-09-24T04:31:00.086ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:56494 req_id = 3b457f03-79bb-4d52-b47c-92a363d6af38 response_code = 200 uri = /baseboard
7952024-09-24T04:31:00.471ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 9121874546550192045 reason = Duration(60s) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7962024-09-24T04:31:00.472ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 16136884810500391095 reason = Duration(60s) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7972024-09-24T04:31:00.937ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7982024-09-24T04:31:00.938ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
7992024-09-24T04:31:01.942ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12313287160412236835 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8002024-09-24T04:31:02.113ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17872697599083210197 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8012024-09-24T04:31:02.218ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2675747394570014431 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8022024-09-24T04:31:04.821ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.359687705s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8032024-09-24T04:31:04.985ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8042024-09-24T04:31:04.985ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60172
8052024-09-24T04:31:04.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8710 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60172 req_id = fee22b3b-b616-4953-8310-52ca4940350c response_code = 200 uri = /timesync
8062024-09-24T04:31:04.998ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152259.3015685, correction: 0.161460534 } file = sled-agent/src/rack_setup/service.rs:660
8072024-09-24T04:31:04.998ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8082024-09-24T04:31:09.412ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14962262056481753924 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8092024-09-24T04:31:09.412ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3091031820373381064 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8102024-09-24T04:31:10.967ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8112024-09-24T04:31:10.967ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8122024-09-24T04:31:11.971ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 12313287160412236835 reason = Duration(60s) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8132024-09-24T04:31:12.141ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 17872697599083210197 reason = Duration(60s) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8142024-09-24T04:31:12.247ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 2675747394570014431 reason = Duration(60s) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8152024-09-24T04:31:13.875ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8162024-09-24T04:31:13.876ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34365
8172024-09-24T04:31:13.885ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9374 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34365 req_id = 90182a0f-3186-47be-b601-901736dc46c2 response_code = 200 uri = /timesync
8182024-09-24T04:31:13.888ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152267.063345, correction: 0.137447938 } file = sled-agent/src/rack_setup/service.rs:660
8192024-09-24T04:31:13.888ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8202024-09-24T04:31:19.441ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 14962262056481753924 reason = Duration(60s) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8212024-09-24T04:31:19.441ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 3091031820373381064 reason = Duration(60s) sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8222024-09-24T04:31:20.314ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8232024-09-24T04:31:20.314ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49721
8242024-09-24T04:31:20.323ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9181 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49721 req_id = a28a17db-78af-4654-9020-5920ed11e837 response_code = 200 uri = /timesync
8252024-09-24T04:31:20.326ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152275.488829, correction: 0.120041072 } file = sled-agent/src/rack_setup/service.rs:660
8262024-09-24T04:31:20.326ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8272024-09-24T04:31:20.997ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8282024-09-24T04:31:20.997ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8292024-09-24T04:31:22.264ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.021133681s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8302024-09-24T04:31:30.201ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40878
8312024-09-24T04:31:30.201ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 29 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40878 req_id = 93bfd683-8aec-4e44-9c48-9fd87a24db69 response_code = 200 uri = /baseboard
8322024-09-24T04:31:31.026ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8332024-09-24T04:31:31.026ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8342024-09-24T04:31:36.909ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8352024-09-24T04:31:36.909ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57923
8362024-09-24T04:31:36.919ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9093 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57923 req_id = 56e2b528-db45-4ac7-b9e2-35a515558a0d response_code = 200 uri = /timesync
8372024-09-24T04:31:36.921ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152291.2522447, correction: 0.075240374 } file = sled-agent/src/rack_setup/service.rs:660
8382024-09-24T04:31:36.921ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8392024-09-24T04:31:40.370ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.30205038s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8402024-09-24T04:31:41.055ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8412024-09-24T04:31:41.055ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8422024-09-24T04:31:51.085ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 468135075039193271 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8432024-09-24T04:31:51.085ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4543063339332335526 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8442024-09-24T04:31:58.948ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8452024-09-24T04:31:58.948ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39279
8462024-09-24T04:31:58.971ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 23060 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39279 req_id = 6c92dbad-d740-41dc-afc9-843dde740d7d response_code = 200 uri = /timesync
8472024-09-24T04:31:58.973ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-09-24T04:31:19.441038487Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 14962262056481753924 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8482024-09-24T04:31:58.973ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-09-24T04:31:12.246996427Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 2675747394570014431 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8492024-09-24T04:31:58.974ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-09-24T04:31:11.971328484Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 12313287160412236835 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8502024-09-24T04:31:58.974ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-09-24T04:31:00.471922165Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 9121874546550192045 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8512024-09-24T04:31:58.974ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-09-24T04:31:19.441371428Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 3091031820373381064 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8522024-09-24T04:31:58.974ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-09-24T04:31:12.141306452Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 17872697599083210197 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8532024-09-24T04:31:58.974ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-09-24T04:31:00.472478930Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 16136884810500391095 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
8542024-09-24T04:31:58.976ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 398108675, ip_addr: 23.186.168.3, stratum: 3, ref_time: 1727152315.6294937, correction: 0.01576791 } file = sled-agent/src/rack_setup/service.rs:660
8552024-09-24T04:31:58.976ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1299
8562024-09-24T04:31:59.005ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 6b582b65-02df-47ca-87c9-8010b32bb48c (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2756a2fa-a955-4f2e-9728-0994c38722e2 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 6d05b27d-c7f2-4261-964b-90b330b71455 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 14c78b5e-89d0-46da-a2db-dc5f845a900e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: c657bb17-8479-480d-b836-9c6e659d7897, kind: Service { id: 14c78b5e-89d0-46da-a2db-dc5f845a900e }, name: Name("ntp-14c78b5e-89d0-46da-a2db-dc5f845a900e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 184, 8])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 26b0fb60-d435-4858-a124-50c3a0e9e29b (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } } } }, OmicronZoneConfig { id: b019a434-decb-4185-abc5-617d2a0c1f81 (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } } } }, OmicronZoneConfig { id: 178901d0-4819-49b7-88e1-99072c2eca92 (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } } } }, OmicronZoneConfig { id: 00a21818-4436-425a-adf5-b8cd1a2591ad (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63332501-9300-44cc-9859-9ceeff24bfdb (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8572024-09-24T04:31:59.005ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61259
8582024-09-24T04:31:59.006ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8592024-09-24T04:31:59.006ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8602024-09-24T04:31:59.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8612024-09-24T04:31:59.024ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8622024-09-24T04:31:59.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8632024-09-24T04:31:59.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }
8642024-09-24T04:31:59.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 26b0fb60-d435-4858-a124-50c3a0e9e29b, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8652024-09-24T04:31:59.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 26b0fb60-d435-4858-a124-50c3a0e9e29b, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8662024-09-24T04:31:59.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b019a434-decb-4185-abc5-617d2a0c1f81, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8672024-09-24T04:31:59.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: b019a434-decb-4185-abc5-617d2a0c1f81, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8682024-09-24T04:31:59.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 178901d0-4819-49b7-88e1-99072c2eca92, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8692024-09-24T04:31:59.167ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 178901d0-4819-49b7-88e1-99072c2eca92, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: Cockroach }, responder: ... }
8702024-09-24T04:31:59.218ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 00a21818-4436-425a-adf5-b8cd1a2591ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8712024-09-24T04:31:59.218ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 00a21818-4436-425a-adf5-b8cd1a2591ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8722024-09-24T04:31:59.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63332501-9300-44cc-9859-9ceeff24bfdb, dataset_name: DatasetName { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8732024-09-24T04:31:59.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 63332501-9300-44cc-9859-9ceeff24bfdb, dataset_name: DatasetName { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8742024-09-24T04:31:59.319ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8752024-09-24T04:31:59.319ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-zones.json file = common/src/ledger.rs:177
8762024-09-24T04:31:59.319ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-zones.json file = common/src/ledger.rs:177
8772024-09-24T04:31:59.319ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8782024-09-24T04:31:59.399ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8792024-09-24T04:31:59.399ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8802024-09-24T04:31:59.399ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8812024-09-24T04:31:59.399ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8822024-09-24T04:31:59.399ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8832024-09-24T04:31:59.491ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b file = illumos-utils/src/zone.rs:316
8842024-09-24T04:31:59.495ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92 file = illumos-utils/src/zone.rs:316
8852024-09-24T04:31:59.500ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb file = illumos-utils/src/zone.rs:316
8862024-09-24T04:31:59.504ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81 file = illumos-utils/src/zone.rs:316
8872024-09-24T04:31:59.508ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad file = illumos-utils/src/zone.rs:316
8882024-09-24T04:31:59.513ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b file = illumos-utils/src/zone.rs:354
8892024-09-24T04:31:59.518ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92 file = illumos-utils/src/zone.rs:354
8902024-09-24T04:31:59.522ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb file = illumos-utils/src/zone.rs:354
8912024-09-24T04:31:59.527ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81 file = illumos-utils/src/zone.rs:354
8922024-09-24T04:31:59.531ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad file = illumos-utils/src/zone.rs:354
8932024-09-24T04:32:00.314ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35071
8942024-09-24T04:32:00.315ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 78 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35071 req_id = 7f184814-16e6-4d0d-ab41-9321e3e1e0f2 response_code = 200 uri = /baseboard
8952024-09-24T04:32:02.469ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='00a21818-4436-425a-adf5-b8cd1a2591ad'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8962024-09-24T04:32:02.469ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
8972024-09-24T04:32:02.475ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='26b0fb60-d435-4858-a124-50c3a0e9e29b'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8982024-09-24T04:32:02.475ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
8992024-09-24T04:32:02.498ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='178901d0-4819-49b7-88e1-99072c2eca92'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9002024-09-24T04:32:02.498ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9012024-09-24T04:32:02.596ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='63332501-9300-44cc-9859-9ceeff24bfdb'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9022024-09-24T04:32:02.596ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9032024-09-24T04:32:02.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 35.034426ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9042024-09-24T04:32:02.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 40.312547ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9052024-09-24T04:32:02.621ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='b019a434-decb-4185-abc5-617d2a0c1f81'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9062024-09-24T04:32:02.622ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9072024-09-24T04:32:02.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 68.045331ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9082024-09-24T04:32:02.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 102.557155ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9092024-09-24T04:32:02.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 148.354395ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9102024-09-24T04:32:02.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 144.320212ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9112024-09-24T04:32:03.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 139.681401ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9122024-09-24T04:32:03.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 171.230348ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9132024-09-24T04:32:03.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 128.942919ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9142024-09-24T04:32:03.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 481.707784ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9152024-09-24T04:32:03.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 52.916194ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9162024-09-24T04:32:03.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 35.420301ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9172024-09-24T04:32:03.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 300.82489ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9182024-09-24T04:32:03.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 69.775983ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9192024-09-24T04:32:03.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 125.943408ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9202024-09-24T04:32:03.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 476.140546ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9212024-09-24T04:32:03.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 647.613695ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9222024-09-24T04:32:03.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 139.680745ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9232024-09-24T04:32:04.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 180.345979ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9242024-09-24T04:32:04.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 945.66414ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9252024-09-24T04:32:04.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 294.199711ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9262024-09-24T04:32:04.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 524.506634ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9272024-09-24T04:32:04.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 1.103376896s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9282024-09-24T04:32:04.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 850.179468ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9292024-09-24T04:32:04.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 735.548766ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9302024-09-24T04:32:04.777ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 11.00935389s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
9312024-09-24T04:32:04.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 1.154131333s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9322024-09-24T04:32:05.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.26298963s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9332024-09-24T04:32:05.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 1.352938991s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9342024-09-24T04:32:05.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 968.603878ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9352024-09-24T04:32:05.511ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 955.775427ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9362024-09-24T04:32:06.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 1.222800741s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9372024-09-24T04:32:06.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 574.218413ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9382024-09-24T04:32:06.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 503.002278ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9392024-09-24T04:32:06.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 686.785037ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9402024-09-24T04:32:06.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 653.734003ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9412024-09-24T04:32:06.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 633.22256ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9422024-09-24T04:32:06.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 646.266353ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9432024-09-24T04:32:07.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 1.371791489s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9442024-09-24T04:32:07.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 706.76063ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9452024-09-24T04:32:07.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 1.21578224s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9462024-09-24T04:32:07.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 663.444616ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9472024-09-24T04:32:07.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 811.503474ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9482024-09-24T04:32:08.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 543.87742ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9492024-09-24T04:32:08.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.23614838s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9502024-09-24T04:32:08.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 543.86743ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9512024-09-24T04:32:08.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 703.484201ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9522024-09-24T04:32:08.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 1.448437308s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9532024-09-24T04:32:08.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 1.257887622s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9542024-09-24T04:32:08.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 735.980485ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9552024-09-24T04:32:09.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 1.041829337s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9562024-09-24T04:32:09.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.035108614s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9572024-09-24T04:32:09.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 903.626814ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9582024-09-24T04:32:09.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 507.50435ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9592024-09-24T04:32:10.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81") failed: Property not found. retry in 723.291223ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9602024-09-24T04:32:10.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad") failed: Property not found. retry in 1.468891378s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9612024-09-24T04:32:10.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b") failed: Property not found. retry in 719.472608ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9622024-09-24T04:32:10.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.152434137s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9632024-09-24T04:32:10.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 833.870413ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9642024-09-24T04:32:11.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 662.558893ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9652024-09-24T04:32:11.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 583.91189ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9662024-09-24T04:32:12.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 1.056987066s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9672024-09-24T04:32:12.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 587.083031ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9682024-09-24T04:32:13.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.413660553s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9692024-09-24T04:32:13.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 1.031737996s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9702024-09-24T04:32:14.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 1.214876687s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9712024-09-24T04:32:14.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.463342242s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9722024-09-24T04:32:15.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 762.684957ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9732024-09-24T04:32:16.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.093935269s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9742024-09-24T04:32:16.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 884.883855ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9752024-09-24T04:32:17.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.251306258s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9762024-09-24T04:32:17.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 524.613888ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9772024-09-24T04:32:17.808ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 519.554024ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9782024-09-24T04:32:18.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 1.079951127s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9792024-09-24T04:32:18.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 981.801457ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9802024-09-24T04:32:19.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 940.830913ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9812024-09-24T04:32:19.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 1.117826845s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9822024-09-24T04:32:20.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92") failed: Property not found. retry in 1.267943928s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9832024-09-24T04:32:20.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb") failed: Property not found. retry in 519.319625ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9842024-09-24T04:32:21.608ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_26b0fb60-d435-4858-a124-50c3a0e9e29b
9852024-09-24T04:32:21.608ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_178901d0-4819-49b7-88e1-99072c2eca92
9862024-09-24T04:32:21.608ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_63332501-9300-44cc-9859-9ceeff24bfdb
9872024-09-24T04:32:21.608ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_b019a434-decb-4185-abc5-617d2a0c1f81
9882024-09-24T04:32:21.608ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_00a21818-4436-425a-adf5-b8cd1a2591ad
9892024-09-24T04:32:21.608ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9902024-09-24T04:32:21.609ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9912024-09-24T04:32:21.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 22587733 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:61259 req_id = 9d79216c-05ab-4da6-9f6f-0ce570dc183b response_code = 204 uri = /omicron-zones
9922024-09-24T04:32:21.644ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35596
9932024-09-24T04:32:21.675ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3713
9942024-09-24T04:32:21.853ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3731
9952024-09-24T04:32:34.716ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3754
9962024-09-24T04:32:34.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 13072025 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:35596 req_id = 30b4f18e-18b5-4322-b98d-bd7735de3cbe response_code = 204 uri = /cockroachdb
9972024-09-24T04:32:34.717ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45902
9982024-09-24T04:32:34.717ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 184 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45902 req_id = 5d6df727-645f-42bf-8aef-4a0275801cf8 response_code = 200 uri = /baseboard
9992024-09-24T04:32:34.722ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 33.996449144s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
10002024-09-24T04:32:34.754ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 6b582b65-02df-47ca-87c9-8010b32bb48c (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2756a2fa-a955-4f2e-9728-0994c38722e2 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 6d05b27d-c7f2-4261-964b-90b330b71455 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 14c78b5e-89d0-46da-a2db-dc5f845a900e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: c657bb17-8479-480d-b836-9c6e659d7897, kind: Service { id: 14c78b5e-89d0-46da-a2db-dc5f845a900e }, name: Name("ntp-14c78b5e-89d0-46da-a2db-dc5f845a900e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 184, 8])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 26b0fb60-d435-4858-a124-50c3a0e9e29b (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } } } }, OmicronZoneConfig { id: b019a434-decb-4185-abc5-617d2a0c1f81 (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } } } }, OmicronZoneConfig { id: 178901d0-4819-49b7-88e1-99072c2eca92 (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } } } }, OmicronZoneConfig { id: 00a21818-4436-425a-adf5-b8cd1a2591ad (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63332501-9300-44cc-9859-9ceeff24bfdb (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: eb18c719-9991-4594-9ed0-1ee13a881361 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 28b4d493-d618-4550-b832-f5a8700dda16 (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 02467522-649f-453a-b95a-5b7b7b4a2458 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 7444bf2e-94a3-4bcb-8a89-2719d1967413 (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } } } }, OmicronZoneConfig { id: f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: a67b216c-59ec-42c9-9875-7fcdb5740448 (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: d3404467-1946-474d-8136-f6d776b403a5 (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: c3d06ea1-0b5c-4478-86f9-447700e6e26e (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External } } } }, OmicronZoneConfig { id: f9dea45d-9df7-4b24-a838-3247c9386378 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External } } } }, OmicronZoneConfig { id: 76e17da7-4e8d-4d25-9aca-040a68999fe9 (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External } } } }, OmicronZoneConfig { id: f906a2fb-29e6-4c4d-897b-0f1e959fdfb4 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External } } } }, OmicronZoneConfig { id: df8455ae-1727-4032-b0a2-1af5db8102b4 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd0d99fb-377a-428b-87b6-80142598d8ca (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a748e81-e24a-442d-9ba3-4573457051e2 (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b3b3030-3910-416f-8f59-f41c59dbb286 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External } } } }, OmicronZoneConfig { id: a80301cb-95d6-434d-8f96-9585464a5296 (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
10012024-09-24T04:32:34.754ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59746
10022024-09-24T04:32:34.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10032024-09-24T04:32:34.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6b582b65-02df-47ca-87c9-8010b32bb48c, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10042024-09-24T04:32:34.779ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10052024-09-24T04:32:34.779ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2756a2fa-a955-4f2e-9728-0994c38722e2, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10062024-09-24T04:32:34.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10072024-09-24T04:32:34.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d05b27d-c7f2-4261-964b-90b330b71455, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: InternalDns }, responder: ... }
10082024-09-24T04:32:34.823ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 26b0fb60-d435-4858-a124-50c3a0e9e29b, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10092024-09-24T04:32:34.823ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 26b0fb60-d435-4858-a124-50c3a0e9e29b, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10102024-09-24T04:32:34.842ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b019a434-decb-4185-abc5-617d2a0c1f81, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10112024-09-24T04:32:34.842ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: b019a434-decb-4185-abc5-617d2a0c1f81, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10122024-09-24T04:32:34.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 178901d0-4819-49b7-88e1-99072c2eca92, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10132024-09-24T04:32:34.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 178901d0-4819-49b7-88e1-99072c2eca92, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: Cockroach }, responder: ... }
10142024-09-24T04:32:34.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 00a21818-4436-425a-adf5-b8cd1a2591ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10152024-09-24T04:32:34.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 00a21818-4436-425a-adf5-b8cd1a2591ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10162024-09-24T04:32:34.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63332501-9300-44cc-9859-9ceeff24bfdb, dataset_name: DatasetName { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10172024-09-24T04:32:34.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 63332501-9300-44cc-9859-9ceeff24bfdb, dataset_name: DatasetName { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10182024-09-24T04:32:34.924ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10192024-09-24T04:32:34.925ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10202024-09-24T04:32:35.051ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eb18c719-9991-4594-9ed0-1ee13a881361, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10212024-09-24T04:32:35.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: eb18c719-9991-4594-9ed0-1ee13a881361, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10222024-09-24T04:32:35.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7444bf2e-94a3-4bcb-8a89-2719d1967413, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
10232024-09-24T04:32:35.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 7444bf2e-94a3-4bcb-8a89-2719d1967413, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10242024-09-24T04:32:35.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c3d06ea1-0b5c-4478-86f9-447700e6e26e, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10252024-09-24T04:32:35.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c3d06ea1-0b5c-4478-86f9-447700e6e26e, dataset_name: DatasetName { pool_name: ZpoolName { id: 036ed427-877d-4d45-ad05-e0fad8a2b1c2 (zpool), kind: External }, kind: Crucible }, responder: ... }
10262024-09-24T04:32:35.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f9dea45d-9df7-4b24-a838-3247c9386378, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10272024-09-24T04:32:35.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f9dea45d-9df7-4b24-a838-3247c9386378, dataset_name: DatasetName { pool_name: ZpoolName { id: 0f5ed56b-52e5-409d-bf51-72e9cba37d51 (zpool), kind: External }, kind: Crucible }, responder: ... }
10282024-09-24T04:32:35.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76e17da7-4e8d-4d25-9aca-040a68999fe9, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10292024-09-24T04:32:35.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 76e17da7-4e8d-4d25-9aca-040a68999fe9, dataset_name: DatasetName { pool_name: ZpoolName { id: d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef (zpool), kind: External }, kind: Crucible }, responder: ... }
10302024-09-24T04:32:35.640ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f906a2fb-29e6-4c4d-897b-0f1e959fdfb4, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10312024-09-24T04:32:35.640ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f906a2fb-29e6-4c4d-897b-0f1e959fdfb4, dataset_name: DatasetName { pool_name: ZpoolName { id: 7a7d72d1-c18a-4053-8666-72377ed40c63 (zpool), kind: External }, kind: Crucible }, responder: ... }
10322024-09-24T04:32:35.751ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8455ae-1727-4032-b0a2-1af5db8102b4, dataset_name: DatasetName { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10332024-09-24T04:32:35.751ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: df8455ae-1727-4032-b0a2-1af5db8102b4, dataset_name: DatasetName { pool_name: ZpoolName { id: cf2b5393-7938-4187-81ff-eefd54b8dca3 (zpool), kind: External }, kind: Crucible }, responder: ... }
10342024-09-24T04:32:35.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd0d99fb-377a-428b-87b6-80142598d8ca, dataset_name: DatasetName { pool_name: ZpoolName { id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10352024-09-24T04:32:35.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bd0d99fb-377a-428b-87b6-80142598d8ca, dataset_name: DatasetName { pool_name: ZpoolName { id: aa8dd6da-a2be-4c64-ab22-604023652cfd (zpool), kind: External }, kind: Crucible }, responder: ... }
10362024-09-24T04:32:35.980ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a748e81-e24a-442d-9ba3-4573457051e2, dataset_name: DatasetName { pool_name: ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10372024-09-24T04:32:35.980ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a748e81-e24a-442d-9ba3-4573457051e2, dataset_name: DatasetName { pool_name: ZpoolName { id: a09f232d-c31e-4961-bb00-74c2dac76055 (zpool), kind: External }, kind: Crucible }, responder: ... }
10382024-09-24T04:32:36.126ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b3b3030-3910-416f-8f59-f41c59dbb286, dataset_name: DatasetName { pool_name: ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10392024-09-24T04:32:36.126ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9b3b3030-3910-416f-8f59-f41c59dbb286, dataset_name: DatasetName { pool_name: ZpoolName { id: f46073e5-0cf7-4937-b820-b3dac0544ced (zpool), kind: External }, kind: Crucible }, responder: ... }
10402024-09-24T04:32:36.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a80301cb-95d6-434d-8f96-9585464a5296, dataset_name: DatasetName { pool_name: ZpoolName { id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10412024-09-24T04:32:36.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a80301cb-95d6-434d-8f96-9585464a5296, dataset_name: DatasetName { pool_name: ZpoolName { id: 5bcd6732-e8a3-402d-925d-d2e58792effe (zpool), kind: External }, kind: Crucible }, responder: ... }
10422024-09-24T04:32:36.357ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10432024-09-24T04:32:36.357ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-zones.json file = common/src/ledger.rs:177
10442024-09-24T04:32:36.357ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-zones.json file = common/src/ledger.rs:177
10452024-09-24T04:32:36.357ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10462024-09-24T04:32:36.478ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
10472024-09-24T04:32:36.504ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
10482024-09-24T04:32:36.504ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10492024-09-24T04:32:36.504ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10502024-09-24T04:32:36.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10512024-09-24T04:32:36.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10522024-09-24T04:32:36.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10532024-09-24T04:32:36.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10542024-09-24T04:32:36.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10552024-09-24T04:32:36.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10562024-09-24T04:32:36.744ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
10572024-09-24T04:32:36.744ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10582024-09-24T04:32:36.744ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10592024-09-24T04:32:36.756ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
10602024-09-24T04:32:36.773ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
10612024-09-24T04:32:36.773ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10622024-09-24T04:32:36.773ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10632024-09-24T04:32:36.773ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10642024-09-24T04:32:36.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10652024-09-24T04:32:36.786ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4 file = illumos-utils/src/zone.rs:316
10662024-09-24T04:32:36.791ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4 file = illumos-utils/src/zone.rs:316
10672024-09-24T04:32:36.796ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458 file = illumos-utils/src/zone.rs:316
10682024-09-24T04:32:36.801ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5 file = illumos-utils/src/zone.rs:316
10692024-09-24T04:32:36.806ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9 file = illumos-utils/src/zone.rs:316
10702024-09-24T04:32:36.811ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286 file = illumos-utils/src/zone.rs:316
10712024-09-24T04:32:36.816ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296 file = illumos-utils/src/zone.rs:316
10722024-09-24T04:32:36.821ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca file = illumos-utils/src/zone.rs:316
10732024-09-24T04:32:36.951ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4 file = illumos-utils/src/zone.rs:354
10742024-09-24T04:32:36.957ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4 file = illumos-utils/src/zone.rs:354
10752024-09-24T04:32:36.962ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458 file = illumos-utils/src/zone.rs:354
10762024-09-24T04:32:36.967ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5 file = illumos-utils/src/zone.rs:354
10772024-09-24T04:32:36.972ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9 file = illumos-utils/src/zone.rs:354
10782024-09-24T04:32:36.977ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286 file = illumos-utils/src/zone.rs:354
10792024-09-24T04:32:36.982ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296 file = illumos-utils/src/zone.rs:354
10802024-09-24T04:32:36.988ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca file = illumos-utils/src/zone.rs:354
10812024-09-24T04:32:36.993ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378 file = illumos-utils/src/zone.rs:316
10822024-09-24T04:32:36.999ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2 file = illumos-utils/src/zone.rs:316
10832024-09-24T04:32:37.005ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448 file = illumos-utils/src/zone.rs:316
10842024-09-24T04:32:37.011ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5 file = illumos-utils/src/zone.rs:316
10852024-09-24T04:32:37.017ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e file = illumos-utils/src/zone.rs:316
10862024-09-24T04:32:37.027ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413 file = illumos-utils/src/zone.rs:316
10872024-09-24T04:32:37.072ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10882024-09-24T04:32:37.072ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378 file = illumos-utils/src/zone.rs:354
10892024-09-24T04:32:37.079ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2 file = illumos-utils/src/zone.rs:354
10902024-09-24T04:32:37.120ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10912024-09-24T04:32:37.120ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448 file = illumos-utils/src/zone.rs:354
10922024-09-24T04:32:37.127ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5 file = illumos-utils/src/zone.rs:354
10932024-09-24T04:32:37.133ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e file = illumos-utils/src/zone.rs:354
10942024-09-24T04:32:37.140ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413 file = illumos-utils/src/zone.rs:354
10952024-09-24T04:32:37.193ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10962024-09-24T04:32:37.257ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10972024-09-24T04:32:37.260ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10982024-09-24T04:32:37.260ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10992024-09-24T04:32:37.323ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 254, 233]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11002024-09-24T04:32:37.323ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
11012024-09-24T04:32:37.325ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11022024-09-24T04:32:37.325ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11032024-09-24T04:32:37.383ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 192, 91]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
11042024-09-24T04:32:37.383ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = external_dns
11052024-09-24T04:32:37.440ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
11062024-09-24T04:32:37.442ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11072024-09-24T04:32:37.442ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11082024-09-24T04:32:37.499ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 253, 24]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11092024-09-24T04:32:37.499ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
11102024-09-24T04:32:37.501ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11112024-09-24T04:32:37.501ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11122024-09-24T04:32:37.677ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 130, 131]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11132024-09-24T04:32:37.677ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
11142024-09-24T04:32:37.685ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
11152024-09-24T04:32:37.685ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
11162024-09-24T04:32:37.685ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
11172024-09-24T04:32:37.685ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
11182024-09-24T04:32:37.687ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11192024-09-24T04:32:37.687ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11202024-09-24T04:32:37.742ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 132, 121]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
11212024-09-24T04:32:37.742ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = external_dns
11222024-09-24T04:32:37.742ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
11232024-09-24T04:32:37.742ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
11242024-09-24T04:32:37.742ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
11252024-09-24T04:32:37.744ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
11262024-09-24T04:32:37.744ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
11272024-09-24T04:32:37.744ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
11282024-09-24T04:32:37.745ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11292024-09-24T04:32:37.746ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
11302024-09-24T04:32:37.747ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11312024-09-24T04:32:37.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11322024-09-24T04:32:37.803ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
11332024-09-24T04:32:37.889ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11342024-09-24T04:32:37.933ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863 file = illumos-utils/src/zone.rs:316
11352024-09-24T04:32:38.003ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f file = illumos-utils/src/zone.rs:316
11362024-09-24T04:32:38.018ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
11372024-09-24T04:32:38.018ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
11382024-09-24T04:32:38.019ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8 file = illumos-utils/src/zone.rs:316
11392024-09-24T04:32:38.033ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863 file = illumos-utils/src/zone.rs:354
11402024-09-24T04:32:38.045ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16 file = illumos-utils/src/zone.rs:316
11412024-09-24T04:32:38.057ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f file = illumos-utils/src/zone.rs:354
11422024-09-24T04:32:38.067ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
11432024-09-24T04:32:38.073ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8 file = illumos-utils/src/zone.rs:354
11442024-09-24T04:32:38.082ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11452024-09-24T04:32:38.204ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16 file = illumos-utils/src/zone.rs:354
11462024-09-24T04:32:38.256ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361 file = illumos-utils/src/zone.rs:316
11472024-09-24T04:32:38.448ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361 file = illumos-utils/src/zone.rs:354
11482024-09-24T04:32:40.053ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11492024-09-24T04:32:40.054ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
11502024-09-24T04:32:40.511ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11512024-09-24T04:32:40.511ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
11522024-09-24T04:32:40.520ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11532024-09-24T04:32:40.520ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_5bcd6732-e8a3-402d-925d-d2e58792effe/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='b0f33852-9323-4e4a-a794-06e019cdb7e5'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11542024-09-24T04:32:40.520ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
11552024-09-24T04:32:40.528ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a8ec4f7f-44a4-4d7c-bc98-cd79a5393858'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11562024-09-24T04:32:40.528ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
11572024-09-24T04:32:40.536ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_aa8dd6da-a2be-4c64-ab22-604023652cfd/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='f5740cfb-603d-4b47-94fb-5987d4f93f5f'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11582024-09-24T04:32:40.536ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a09f232d-c31e-4961-bb00-74c2dac76055/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='12c48a66-0145-4bbf-be30-ac4d0db6cbf7'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11592024-09-24T04:32:40.537ZINFOSledAgent (ServiceManager): Booting oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
11602024-09-24T04:32:40.544ZINFOSledAgent (ServiceManager): Booting oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
11612024-09-24T04:32:40.550ZINFOSledAgent (ServiceManager): Booting oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
11622024-09-24T04:32:40.575ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7a7d72d1-c18a-4053-8666-72377ed40c63/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='71f34c7a-8d7d-4001-b89f-b2c741ea8c60'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11632024-09-24T04:32:40.575ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
11642024-09-24T04:32:40.592ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_cf2b5393-7938-4187-81ff-eefd54b8dca3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='9e0889a2-50e2-4406-8785-ea2a5db10069'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11652024-09-24T04:32:40.592ZINFOSledAgent (ServiceManager): Booting oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
11662024-09-24T04:32:40.605ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_036ed427-877d-4d45-ad05-e0fad8a2b1c2/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='067bbcf6-6262-4098-91ee-8e029013f51e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11672024-09-24T04:32:40.605ZINFOSledAgent (ServiceManager): Booting oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
11682024-09-24T04:32:40.624ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_0f5ed56b-52e5-409d-bf51-72e9cba37d51/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='31604faa-252d-4c2c-af43-02d2d4696519'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11692024-09-24T04:32:40.624ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
11702024-09-24T04:32:40.631ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f46073e5-0cf7-4937-b820-b3dac0544ced/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='698b5093-5889-4c5c-bea6-a3c41079a9d8'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11712024-09-24T04:32:40.631ZINFOSledAgent (ServiceManager): Booting oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
11722024-09-24T04:32:40.757ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='02467522-649f-453a-b95a-5b7b7b4a2458'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11732024-09-24T04:32:40.757ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
11742024-09-24T04:32:44.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 68.402448ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
11752024-09-24T04:32:44.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 30.394912ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
11762024-09-24T04:32:44.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 55.808316ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
11772024-09-24T04:32:44.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 52.718317ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
11782024-09-24T04:32:44.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 39.228525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
11792024-09-24T04:32:44.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 58.503254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
11802024-09-24T04:32:44.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 57.89135ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
11812024-09-24T04:32:44.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 31.914363ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
11822024-09-24T04:32:44.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 149.355742ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
11832024-09-24T04:32:44.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 69.667686ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
11842024-09-24T04:32:44.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 106.530637ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
11852024-09-24T04:32:44.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 55.075568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
11862024-09-24T04:32:44.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 56.568509ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
11872024-09-24T04:32:44.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 57.944611ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
11882024-09-24T04:32:44.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 103.457082ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
11892024-09-24T04:32:44.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 110.685255ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
11902024-09-24T04:32:44.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 82.608038ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
11912024-09-24T04:32:44.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 59.260961ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
11922024-09-24T04:32:44.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 85.108245ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
11932024-09-24T04:32:44.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 69.76449ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
11942024-09-24T04:32:44.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 146.366631ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
11952024-09-24T04:32:44.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 143.433359ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
11962024-09-24T04:32:44.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 118.926307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
11972024-09-24T04:32:44.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 52.684286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
11982024-09-24T04:32:44.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 208.807816ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
11992024-09-24T04:32:44.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 272.420784ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12002024-09-24T04:32:44.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 260.761901ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12012024-09-24T04:32:44.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 224.126565ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12022024-09-24T04:32:44.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 135.248663ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12032024-09-24T04:32:44.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 180.912417ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12042024-09-24T04:32:44.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 184.591964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12052024-09-24T04:32:44.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 267.995833ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12062024-09-24T04:32:44.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 272.05022ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12072024-09-24T04:32:44.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 284.543695ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12082024-09-24T04:32:44.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 144.070988ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12092024-09-24T04:32:44.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 171.859106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12102024-09-24T04:32:44.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 404.160335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12112024-09-24T04:32:44.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 315.298606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12122024-09-24T04:32:44.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 473.251994ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12132024-09-24T04:32:44.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 297.389266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12142024-09-24T04:32:44.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 522.986525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12152024-09-24T04:32:44.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 311.514402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12162024-09-24T04:32:44.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 552.487175ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12172024-09-24T04:32:44.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 396.897357ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12182024-09-24T04:32:44.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 581.758252ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12192024-09-24T04:32:44.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 474.57322ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12202024-09-24T04:32:44.770ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 475.208474ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12212024-09-24T04:32:44.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 347.111565ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12222024-09-24T04:32:44.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 1.121414725s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12232024-09-24T04:32:44.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 441.053807ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12242024-09-24T04:32:45.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.098299435s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12252024-09-24T04:32:45.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 661.499148ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12262024-09-24T04:32:45.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 924.927787ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12272024-09-24T04:32:45.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 794.298262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12282024-09-24T04:32:45.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 702.70756ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12292024-09-24T04:32:45.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 447.661155ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12302024-09-24T04:32:45.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 776.892751ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12312024-09-24T04:32:45.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 871.395211ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12322024-09-24T04:32:45.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 551.248132ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12332024-09-24T04:32:45.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 885.556053ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12342024-09-24T04:32:45.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 550.472433ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12352024-09-24T04:32:45.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 925.339862ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12362024-09-24T04:32:45.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.390192434s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12372024-09-24T04:32:45.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 826.338405ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12382024-09-24T04:32:45.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.025079497s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12392024-09-24T04:32:45.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.41785497s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12402024-09-24T04:32:45.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 1.163525492s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12412024-09-24T04:32:46.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 507.541323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12422024-09-24T04:32:46.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.296228847s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12432024-09-24T04:32:46.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 569.100393ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12442024-09-24T04:32:46.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.420020623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12452024-09-24T04:32:46.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 1.169284479s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12462024-09-24T04:32:46.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.110793819s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12472024-09-24T04:32:46.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 1.131251085s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12482024-09-24T04:32:46.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 1.456151917s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12492024-09-24T04:32:46.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 667.774023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12502024-09-24T04:32:46.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.312107214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12512024-09-24T04:32:46.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.25141321s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12522024-09-24T04:32:47.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 874.103352ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12532024-09-24T04:32:47.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 1.38952813s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12542024-09-24T04:32:47.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 1.144871735s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12552024-09-24T04:32:47.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 1.099738791s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12562024-09-24T04:32:47.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.030296536s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12572024-09-24T04:32:47.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.012485722s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12582024-09-24T04:32:47.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.140987438s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12592024-09-24T04:32:47.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 757.911558ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12602024-09-24T04:32:47.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 805.771397ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12612024-09-24T04:32:47.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.189043923s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12622024-09-24T04:32:48.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.445723442s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12632024-09-24T04:32:48.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 518.698976ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12642024-09-24T04:32:48.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.36440609s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12652024-09-24T04:32:48.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.337392273s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12662024-09-24T04:32:48.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 616.233393ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12672024-09-24T04:32:48.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.086049699s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12682024-09-24T04:32:48.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 829.364322ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12692024-09-24T04:32:48.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 1.416502148s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12702024-09-24T04:32:48.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 1.042244136s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12712024-09-24T04:32:48.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.370551644s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12722024-09-24T04:32:48.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 747.976037ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12732024-09-24T04:32:48.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 547.667387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12742024-09-24T04:32:49.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.22905458s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12752024-09-24T04:32:49.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 600.521112ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12762024-09-24T04:32:49.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.124169867s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12772024-09-24T04:32:49.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 1.266127537s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12782024-09-24T04:32:49.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 1.253518945s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12792024-09-24T04:32:49.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.277436602s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12802024-09-24T04:32:49.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 811.043933ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12812024-09-24T04:32:49.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 559.793913ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12822024-09-24T04:32:49.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 876.668185ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12832024-09-24T04:32:49.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 782.634448ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12842024-09-24T04:32:49.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 904.611209ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12852024-09-24T04:32:49.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 553.942465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12862024-09-24T04:32:49.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 840.042106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12872024-09-24T04:32:50.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 746.766347ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
12882024-09-24T04:32:50.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 576.571043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
12892024-09-24T04:32:50.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 656.920783ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
12902024-09-24T04:32:50.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.308004763s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
12912024-09-24T04:32:50.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 876.278188ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
12922024-09-24T04:32:50.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 984.516208ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
12932024-09-24T04:32:50.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 769.384515ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
12942024-09-24T04:32:50.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 820.979717ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
12952024-09-24T04:32:50.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 1.329576019s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
12962024-09-24T04:32:50.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 1.288807463s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
12972024-09-24T04:32:50.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 927.636698ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
12982024-09-24T04:32:50.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.133720256s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
12992024-09-24T04:32:50.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.301595283s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
13002024-09-24T04:32:50.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.372777679s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
13012024-09-24T04:32:51.078ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.329742499s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
13022024-09-24T04:32:51.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.3728629s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
13032024-09-24T04:32:51.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 926.139181ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
13042024-09-24T04:32:51.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 740.593333ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
13052024-09-24T04:32:51.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 591.558974ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
13062024-09-24T04:32:51.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 603.799301ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
13072024-09-24T04:32:51.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 785.710492ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
13082024-09-24T04:32:51.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.021064959s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
13092024-09-24T04:32:51.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 1.204429666s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
13102024-09-24T04:32:52.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 875.838489ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
13112024-09-24T04:32:52.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 823.622836ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
13122024-09-24T04:32:52.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 1.284233831s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
13132024-09-24T04:32:52.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.142706921s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
13142024-09-24T04:32:52.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 1.248798654s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
13152024-09-24T04:32:52.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 715.151491ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
13162024-09-24T04:32:52.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 709.143592ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
13172024-09-24T04:32:52.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 679.47193ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
13182024-09-24T04:32:52.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.359891741s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
13192024-09-24T04:32:52.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 879.638972ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
13202024-09-24T04:32:52.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 67.440386ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13212024-09-24T04:32:52.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 736.647508ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
13222024-09-24T04:32:52.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 895.74133ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
13232024-09-24T04:32:53.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 109.550311ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13242024-09-24T04:32:53.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.382266087s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
13252024-09-24T04:32:53.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.093976206s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
13262024-09-24T04:32:53.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.18902881s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
13272024-09-24T04:32:53.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 542.941911ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
13282024-09-24T04:32:53.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 244.988152ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13292024-09-24T04:32:53.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 631.408864ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
13302024-09-24T04:32:53.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.322739161s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
13312024-09-24T04:32:53.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 468.544131ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13322024-09-24T04:32:53.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 1.277339795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
13332024-09-24T04:32:53.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 1.359637133s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
13342024-09-24T04:32:53.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.420700446s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
13352024-09-24T04:32:53.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 1.161137448s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
13362024-09-24T04:32:53.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.302723732s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
13372024-09-24T04:32:53.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 907.912079ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
13382024-09-24T04:32:53.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 784.868712ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
13392024-09-24T04:32:53.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 856.853874ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13402024-09-24T04:32:53.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.16597828s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
13412024-09-24T04:32:54.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.081419381s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
13422024-09-24T04:32:54.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 866.84407ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
13432024-09-24T04:32:54.301ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13442024-09-24T04:32:54.302ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
13452024-09-24T04:32:54.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 894.095382ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
13462024-09-24T04:32:54.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 57.27155ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
13472024-09-24T04:32:54.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 143.792828ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
13482024-09-24T04:32:54.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 1.256569736s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
13492024-09-24T04:32:54.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 523.611309ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
13502024-09-24T04:32:54.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 1.419426602s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
13512024-09-24T04:32:54.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 593.933038ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
13522024-09-24T04:32:54.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 525.512155ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
13532024-09-24T04:32:54.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 854.842758ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13542024-09-24T04:32:54.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 233.965011ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
13552024-09-24T04:32:54.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 1.201042233s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
13562024-09-24T04:32:55.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 627.352306ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
13572024-09-24T04:32:55.025ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13582024-09-24T04:32:55.025ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
13592024-09-24T04:32:55.034ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13602024-09-24T04:32:55.034ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
13612024-09-24T04:32:55.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 635.179789ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
13622024-09-24T04:32:55.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 790.477324ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
13632024-09-24T04:32:55.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.027701831s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
13642024-09-24T04:32:55.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 329.001432ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
13652024-09-24T04:32:55.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.280802929s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
13662024-09-24T04:32:55.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 927.525774ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
13672024-09-24T04:32:55.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 53.411597ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
13682024-09-24T04:32:55.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 64.877175ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
13692024-09-24T04:32:55.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.172146553s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
13702024-09-24T04:32:55.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 77.318465ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
13712024-09-24T04:32:55.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 821.283459ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
13722024-09-24T04:32:55.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 75.242756ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
13732024-09-24T04:32:55.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 943.607657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
13742024-09-24T04:32:55.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 147.008815ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
13752024-09-24T04:32:55.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 264.951091ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
13762024-09-24T04:32:55.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 521.993692ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
13772024-09-24T04:32:55.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 793.582909ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
13782024-09-24T04:32:55.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 480.38909ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
13792024-09-24T04:32:55.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.036927118s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
13802024-09-24T04:32:55.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 900.755037ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13812024-09-24T04:32:55.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 243.734825ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
13822024-09-24T04:32:55.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 1.459471491s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
13832024-09-24T04:32:55.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.24575024s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
13842024-09-24T04:32:56.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 1.018320523s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
13852024-09-24T04:32:56.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 618.928124ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
13862024-09-24T04:32:56.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 1.158801392s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
13872024-09-24T04:32:56.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 706.65679ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
13882024-09-24T04:32:56.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 582.822378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
13892024-09-24T04:32:56.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 736.706525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
13902024-09-24T04:32:56.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 902.332587ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
13912024-09-24T04:32:56.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 768.452886ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
13922024-09-24T04:32:56.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 521.453325ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
13932024-09-24T04:32:56.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.073026022s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
13942024-09-24T04:32:56.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 806.78071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
13952024-09-24T04:32:56.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.264800111s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
13962024-09-24T04:32:56.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 1.199475356s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
13972024-09-24T04:32:56.674ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13982024-09-24T04:32:56.674ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13992024-09-24T04:32:56.674ZINFOSledAgent (ServiceManager): Booting oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14002024-09-24T04:32:56.683ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14012024-09-24T04:32:56.683ZINFOSledAgent (ServiceManager): Booting oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14022024-09-24T04:32:56.690ZINFOSledAgent (ServiceManager): Booting oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14032024-09-24T04:32:56.770ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 1.283362787s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
14042024-09-24T04:32:56.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 802.964768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
14052024-09-24T04:32:56.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 639.723484ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
14062024-09-24T04:32:56.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 51.825324ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14072024-09-24T04:32:56.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 879.729097ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
14082024-09-24T04:32:56.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 916.344252ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
14092024-09-24T04:32:56.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 31.288951ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14102024-09-24T04:32:56.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 26.321338ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14112024-09-24T04:32:57.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 84.074583ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14122024-09-24T04:32:57.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 949.386568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
14132024-09-24T04:32:57.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 131.600929ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14142024-09-24T04:32:57.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 241.068993ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14152024-09-24T04:32:57.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 113.484234ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14162024-09-24T04:32:57.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 1.171276846s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
14172024-09-24T04:32:57.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.104524169s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
14182024-09-24T04:32:57.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 540.561181ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
14192024-09-24T04:32:57.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 810.179413ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
14202024-09-24T04:32:57.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 1.336082969s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
14212024-09-24T04:32:57.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 903.627438ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
14222024-09-24T04:32:57.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 1.039414269s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
14232024-09-24T04:32:57.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 136.410754ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14242024-09-24T04:32:57.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 560.260028ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14252024-09-24T04:32:57.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 912.819308ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
14262024-09-24T04:32:57.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 249.855335ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14272024-09-24T04:32:57.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 867.004675ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
14282024-09-24T04:32:57.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 844.210522ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
14292024-09-24T04:32:57.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 980.201637ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
14302024-09-24T04:32:57.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.372418315s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
14312024-09-24T04:32:57.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 867.154845ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
14322024-09-24T04:32:57.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 541.97706ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14332024-09-24T04:32:57.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 1.362991116s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
14342024-09-24T04:32:58.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 828.387695ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
14352024-09-24T04:32:58.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 695.468214ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
14362024-09-24T04:32:58.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 891.247309ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
14372024-09-24T04:32:58.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 390.224503ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14382024-09-24T04:32:58.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 986.956136ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14392024-09-24T04:32:58.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 1.441335063s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
14402024-09-24T04:32:58.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 915.505217ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
14412024-09-24T04:32:58.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 612.554407ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14422024-09-24T04:32:58.541ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 604.417695ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14432024-09-24T04:32:58.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 957.062732ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
14442024-09-24T04:32:58.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 1.263938814s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
14452024-09-24T04:32:58.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 877.46339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
14462024-09-24T04:32:58.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.409815041s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
14472024-09-24T04:32:58.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 700.490066ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
14482024-09-24T04:32:58.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.161604193s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
14492024-09-24T04:32:58.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.007490435s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
14502024-09-24T04:32:58.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 1.329065879s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
14512024-09-24T04:32:58.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 998.407206ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
14522024-09-24T04:32:58.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 1.304118349s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
14532024-09-24T04:32:58.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 816.478186ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
14542024-09-24T04:32:58.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 774.961089ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
14552024-09-24T04:32:59.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 542.576333ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14562024-09-24T04:32:59.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 819.798496ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14572024-09-24T04:32:59.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 1.056469658s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14582024-09-24T04:32:59.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.147427116s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
14592024-09-24T04:32:59.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 905.986344ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
14602024-09-24T04:32:59.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.277934217s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
14612024-09-24T04:32:59.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 557.979649ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
14622024-09-24T04:32:59.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 1.2518393s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
14632024-09-24T04:32:59.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 508.440427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
14642024-09-24T04:32:59.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 1.443472788s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14652024-09-24T04:32:59.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.017727091s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
14662024-09-24T04:32:59.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 641.947919ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
14672024-09-24T04:32:59.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 532.147093ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
14682024-09-24T04:32:59.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458") failed: Property not found. retry in 1.470940417s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
14692024-09-24T04:32:59.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 1.056698406s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
14702024-09-24T04:32:59.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 1.137335533s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
14712024-09-24T04:32:59.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 727.107797ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
14722024-09-24T04:33:00.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 1.3562177s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14732024-09-24T04:33:00.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 772.112561ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
14742024-09-24T04:33:00.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 521.85745ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
14752024-09-24T04:33:00.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 747.656596ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
14762024-09-24T04:33:00.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 967.269395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
14772024-09-24T04:33:00.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 1.438918849s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
14782024-09-24T04:33:00.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 889.559463ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
14792024-09-24T04:33:00.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 1.229937092s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
14802024-09-24T04:33:00.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 1.02471653s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
14812024-09-24T04:33:00.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 571.15208ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
14822024-09-24T04:33:00.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 1.393951635s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
14832024-09-24T04:33:00.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296") failed: Property not found. retry in 1.469832801s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
14842024-09-24T04:33:00.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.311418252s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
14852024-09-24T04:33:00.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5") failed: Property not found. retry in 1.186059946s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
14862024-09-24T04:33:00.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5") failed: Property not found. retry in 802.939239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
14872024-09-24T04:33:00.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 521.285367ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
14882024-09-24T04:33:00.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 794.312641ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
14892024-09-24T04:33:00.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.152853916s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
14902024-09-24T04:33:00.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9") failed: Property not found. retry in 1.357523813s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
14912024-09-24T04:33:01.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 959.538068ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
14922024-09-24T04:33:01.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 1.317130588s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
14932024-09-24T04:33:01.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 849.567428ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
14942024-09-24T04:33:01.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 791.035699ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
14952024-09-24T04:33:01.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 633.216825ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
14962024-09-24T04:33:01.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286") failed: Property not found. retry in 1.348252091s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
14972024-09-24T04:33:01.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 666.864382ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
14982024-09-24T04:33:01.398ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 843.04154ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
14992024-09-24T04:33:01.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 907.568304ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
15002024-09-24T04:33:01.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2") failed: Property not found. retry in 1.276999415s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
15012024-09-24T04:33:01.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 855.435887ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
15022024-09-24T04:33:01.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4") failed: Property not found. retry in 536.087259ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
15032024-09-24T04:33:01.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca") failed: Property not found. retry in 519.654203ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
15042024-09-24T04:33:01.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378") failed: Property not found. retry in 975.728426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
15052024-09-24T04:33:02.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 581.659102ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
15062024-09-24T04:33:02.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4") failed: Property not found. retry in 1.325893539s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
15072024-09-24T04:33:02.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448") failed: Property not found. retry in 1.149165236s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
15082024-09-24T04:33:02.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 908.42998ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
15092024-09-24T04:33:02.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 804.98874ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
15102024-09-24T04:33:02.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 839.877149ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15112024-09-24T04:33:02.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 636.703374ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
15122024-09-24T04:33:02.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 817.944318ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
15132024-09-24T04:33:02.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 1.103748653s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
15142024-09-24T04:33:02.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 513.126177ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
15152024-09-24T04:33:02.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 609.404575ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
15162024-09-24T04:33:03.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 990.711671ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
15172024-09-24T04:33:03.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e") failed: Property not found. retry in 1.360764412s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
15182024-09-24T04:33:03.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 599.24347ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15192024-09-24T04:33:03.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 504.079311ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
15202024-09-24T04:33:03.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 890.788849ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
15212024-09-24T04:33:03.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 842.563789ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
15222024-09-24T04:33:03.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 946.16142ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
15232024-09-24T04:33:03.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 947.10851ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
15242024-09-24T04:33:03.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 583.637191ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15252024-09-24T04:33:04.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 1.486711812s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
15262024-09-24T04:33:04.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 912.58831ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
15272024-09-24T04:33:04.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 1.049997471s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15282024-09-24T04:33:04.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 1.058869295s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
15292024-09-24T04:33:04.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f") failed: Property not found. retry in 1.306834201s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
15302024-09-24T04:33:04.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 711.249443ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
15312024-09-24T04:33:04.802ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:55220
15322024-09-24T04:33:04.804ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 659 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:55220 req_id = daf7238a-7ccf-4c3b-97bb-bbcc280b62b5 response_code = 200 uri = /baseboard
15332024-09-24T04:33:05.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863") failed: Property not found. retry in 853.847254ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
15342024-09-24T04:33:05.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361") failed: Property not found. retry in 1.113613873s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
15352024-09-24T04:33:05.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 652.304005ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15362024-09-24T04:33:05.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8") failed: Property not found. retry in 1.033861215s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
15372024-09-24T04:33:05.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16") failed: Property not found. retry in 883.269954ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
15382024-09-24T04:33:06.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 1.155018093s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15392024-09-24T04:33:07.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 619.622742ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15402024-09-24T04:33:07.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 1.047219975s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15412024-09-24T04:33:08.779ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 60.872109488s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
15422024-09-24T04:33:08.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 620.201015ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15432024-09-24T04:33:09.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 503.178681ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15442024-09-24T04:33:10.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 994.021251ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15452024-09-24T04:33:11.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 974.214943ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15462024-09-24T04:33:12.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 1.305412848s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15472024-09-24T04:33:13.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 1.379294015s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15482024-09-24T04:33:14.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 710.445806ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15492024-09-24T04:33:15.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413") failed: Property not found. retry in 785.591505ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15502024-09-24T04:33:16.298ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_9c4a6c7c-d0f3-4040-893d-c86718cd5863
15512024-09-24T04:33:16.298ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_f906a2fb-29e6-4c4d-897b-0f1e959fdfb4
15522024-09-24T04:33:16.298ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_df8455ae-1727-4032-b0a2-1af5db8102b4
15532024-09-24T04:33:16.298ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_28b4d493-d618-4550-b832-f5a8700dda16
15542024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_oximeter_02467522-649f-453a-b95a-5b7b7b4a2458
15552024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_d3404467-1946-474d-8136-f6d776b403a5
15562024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_76e17da7-4e8d-4d25-9aca-040a68999fe9
15572024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_9b3b3030-3910-416f-8f59-f41c59dbb286
15582024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_a80301cb-95d6-434d-8f96-9585464a5296
15592024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_bd0d99fb-377a-428b-87b6-80142598d8ca
15602024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_f9dea45d-9df7-4b24-a838-3247c9386378
15612024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_6a748e81-e24a-442d-9ba3-4573457051e2
15622024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_external_dns_3a3c9bd7-6179-4647-88be-743dbba6ce2f
15632024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_external_dns_eb18c719-9991-4594-9ed0-1ee13a881361
15642024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_a67b216c-59ec-42c9-9875-7fcdb5740448
15652024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_f00c2874-c0e6-4d1a-8f02-2a7f6fa6b5f5
15662024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_c3d06ea1-0b5c-4478-86f9-447700e6e26e
15672024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_19d1b3bf-4d38-489c-a831-139faa2845a8
15682024-09-24T04:33:16.299ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_clickhouse_7444bf2e-94a3-4bcb-8a89-2719d1967413
15692024-09-24T04:33:16.300ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15702024-09-24T04:33:16.301ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15712024-09-24T04:33:16.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 41549921 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59746 req_id = 9fccce9c-73c5-40cb-b97f-992c1b152f79 response_code = 204 uri = /omicron-zones
15722024-09-24T04:33:16.313ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1322
15732024-09-24T04:33:16.314ZINFOSledAgent (RSS): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
15742024-09-24T04:33:16.314ZINFOSledAgent (RSS): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
15752024-09-24T04:33:16.318ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
15762024-09-24T04:33:16.318ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:720
15772024-09-24T04:33:16.351ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:887
15782024-09-24T04:33:16.352ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15792024-09-24T04:33:16.466ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15802024-09-24T04:33:16.553ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15812024-09-24T04:33:16.671ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15822024-09-24T04:33:16.834ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15832024-09-24T04:33:16.983ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15842024-09-24T04:33:17.224ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15852024-09-24T04:33:17.578ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15862024-09-24T04:33:17.802ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15872024-09-24T04:33:18.096ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15882024-09-24T04:33:18.770ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15892024-09-24T04:33:19.439ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15902024-09-24T04:33:20.030ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15912024-09-24T04:33:20.132ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
15922024-09-24T04:33:20.813ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
15932024-09-24T04:33:21.239ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15942024-09-24T04:33:21.292ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
15952024-09-24T04:33:22.258ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
15962024-09-24T04:33:22.295ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
15972024-09-24T04:33:22.813ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
15982024-09-24T04:33:23.298ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
15992024-09-24T04:33:24.107ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/052f3529-69d0-4ca1-873b-88ee5f382b59/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16002024-09-24T04:33:24.301ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
16012024-09-24T04:33:24.812ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
16022024-09-24T04:33:25.304ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/223823d0-6342-4be2-8483-3bbd7919ea67): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
16032024-09-24T04:33:26.518ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "8011288b-33e3-49b4-b204-88d20f35bda1", "content-length": "173", "date": "Tue, 24 Sep 2024 04:33:26 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"052f3529-69d0-4ca1-873b-88ee5f382b59\"", request_id: "8011288b-33e3-49b4-b204-88d20f35bda1" } file = sled-agent/src/rack_setup/service.rs:949
16042024-09-24T04:33:28.909ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
16052024-09-24T04:33:29.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52289
16062024-09-24T04:33:29.073ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16072024-09-24T04:33:29.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 226642 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52289 req_id = eaa624f9-2b5b-4e5f-9a3a-5ab5eb449fa8 response_code = 200 uri = /inventory
16082024-09-24T04:33:29.300ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16092024-09-24T04:33:29.300ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-zones.json file = common/src/ledger.rs:177
16102024-09-24T04:33:29.301ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-zones.json file = common/src/ledger.rs:177
16112024-09-24T04:33:29.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52289 req_id = 240dd661-081e-4409-9373-f7e832e0857a response_code = 200 uri = /omicron-zones
16122024-09-24T04:33:29.984ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61538
16132024-09-24T04:33:29.986ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16142024-09-24T04:33:29.986ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16152024-09-24T04:33:29.986ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16162024-09-24T04:33:29.986ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16172024-09-24T04:33:29.986ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
16182024-09-24T04:33:29.986ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16192024-09-24T04:33:29.986ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16202024-09-24T04:33:29.986ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2229 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61538 req_id = 282d0c0b-7f09-455c-9f52-6e1a448ca408 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16212024-09-24T04:33:29.989ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:959
16222024-09-24T04:33:30.289ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39239
16232024-09-24T04:33:30.289ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16242024-09-24T04:33:30.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 223202 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39239 req_id = c5834fd7-0880-47a1-a052-1cae024f3305 response_code = 200 uri = /inventory
16252024-09-24T04:33:30.513ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16262024-09-24T04:33:30.513ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/config/omicron-zones.json file = common/src/ledger.rs:177
16272024-09-24T04:33:30.513ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/config/omicron-zones.json file = common/src/ledger.rs:177
16282024-09-24T04:33:30.513ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 875 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39239 req_id = 65d1d4f6-4c83-41eb-83e1-030f90260b2b response_code = 200 uri = /omicron-zones
16292024-09-24T04:33:31.717ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48225
16302024-09-24T04:33:31.718ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16312024-09-24T04:33:31.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16322024-09-24T04:33:31.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16332024-09-24T04:33:31.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16342024-09-24T04:33:31.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
16352024-09-24T04:33:31.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16362024-09-24T04:33:31.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16372024-09-24T04:33:31.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 312 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48225 req_id = 21e78fba-23ee-48a2-aa03-972b17c9e9c6 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16382024-09-24T04:33:31.824ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45643
16392024-09-24T04:33:31.824ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16402024-09-24T04:33:31.824ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16412024-09-24T04:33:31.824ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16422024-09-24T04:33:31.824ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16432024-09-24T04:33:31.824ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
16442024-09-24T04:33:31.824ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16452024-09-24T04:33:31.824ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16462024-09-24T04:33:31.824ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 355 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45643 req_id = acf237c8-5840-4be6-b42a-d0cbbf066e08 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16472024-09-24T04:33:31.875ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50978
16482024-09-24T04:33:31.876ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16492024-09-24T04:33:31.876ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16502024-09-24T04:33:31.876ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16512024-09-24T04:33:31.876ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16522024-09-24T04:33:31.876ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
16532024-09-24T04:33:31.876ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16542024-09-24T04:33:31.876ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16552024-09-24T04:33:31.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 791 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50978 req_id = 8b7f2a67-0563-4512-a9e9-6a20436a1376 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16562024-09-24T04:33:34.848ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:38562
16572024-09-24T04:33:34.848ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:38562 req_id = 54d1ccc9-ddc4-4e45-9131-7c4644fc9c23 response_code = 200 uri = /baseboard
16582024-09-24T04:33:36.416ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63895
16592024-09-24T04:33:36.417ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63895 req_id = dcc0ccd1-5d9d-4949-a4de-f17c6cc4307f response_code = 200 uri = /vpc-routes
16602024-09-24T04:33:36.492ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1823 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63895 req_id = a9fb95fd-80a7-43b6-bacd-98f6d6b690cc response_code = 204 uri = /vpc-routes
16612024-09-24T04:33:57.582ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44842
16622024-09-24T04:33:57.582ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44842 req_id = 6bd0dcfc-7a41-48f4-aaf9-3a0c8643a980 response_code = 200 uri = /vpc-routes
16632024-09-24T04:33:57.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44842 req_id = f15e8dd6-a857-49a0-8003-bc0583eacbb1 response_code = 204 uri = /vpc-routes
16642024-09-24T04:33:57.770ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42240
16652024-09-24T04:33:57.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42240 req_id = af47a69a-da19-43b7-9dc0-397133671276 response_code = 200 uri = /v2p
16662024-09-24T04:33:57.823ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36909
16672024-09-24T04:33:57.824ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16682024-09-24T04:33:57.889ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16692024-09-24T04:33:57.944ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 121242 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36909 req_id = fe45cab2-cfd6-4908-afef-8ad984dbee78 response_code = 204 uri = /switch-ports
16702024-09-24T04:33:57.945ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1084 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36909 req_id = a1932cd4-9de5-4aea-9ffc-27dde743dd29 response_code = 200 uri = /network-bootstore-config
16712024-09-24T04:33:58.104ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
16722024-09-24T04:33:58.105ZINFOSledAgent (bootstore): Writing ledger to /pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
16732024-09-24T04:33:58.106ZINFOSledAgent (bootstore): Writing ledger to /pool/int/1933fe24-deab-4311-a300-a09ef207e36a/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
16742024-09-24T04:33:58.107ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
16752024-09-24T04:33:58.107ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3530 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36909 req_id = fa03e11f-0d2d-4a0c-8fc0-572acfb6ef53 response_code = 204 uri = /network-bootstore-config
16762024-09-24T04:33:58.392ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36536
16772024-09-24T04:33:58.393ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 805 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36536 req_id = 95a7ee62-e902-4a1a-9de5-6b9173f7617b response_code = 200 uri = /vpc-routes
16782024-09-24T04:33:58.422ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49717
16792024-09-24T04:33:58.422ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49717 req_id = a1b5f886-4f32-4730-af5f-e06d5cad18cd response_code = 200 uri = /vpc-routes
16802024-09-24T04:33:58.434ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49717 req_id = baf1c6c6-45c3-45f8-8b98-585677a2d3f4 response_code = 204 uri = /vpc-routes
16812024-09-24T04:33:58.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61025
16822024-09-24T04:33:58.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 621 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61025 req_id = e8274ae8-0bf1-4b58-bafb-e4fa17f0a6ed response_code = 200 uri = /v2p
16832024-09-24T04:33:58.452ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36536 req_id = 70454b9a-2421-40f3-beb6-371bde507b0e response_code = 204 uri = /vpc-routes
16842024-09-24T04:33:58.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48759
16852024-09-24T04:33:58.486ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16862024-09-24T04:33:58.525ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16872024-09-24T04:33:58.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93810 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48759 req_id = c5039074-55d7-4797-9cb4-d744bd220c33 response_code = 204 uri = /switch-ports
16882024-09-24T04:33:58.580ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47190
16892024-09-24T04:33:58.580ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54246
16902024-09-24T04:33:58.580ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16912024-09-24T04:33:58.580ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 317 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47190 req_id = 521ac0be-e612-4b66-b0bd-f23942ebf123 response_code = 200 uri = /v2p
16922024-09-24T04:33:58.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 614 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48759 req_id = d5b73483-8a57-4f83-9825-c554fa54833e response_code = 200 uri = /network-bootstore-config
16932024-09-24T04:33:58.618ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16942024-09-24T04:33:58.676ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96197 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54246 req_id = 7f26f287-a156-4dd0-aae6-6493f12d49e9 response_code = 204 uri = /switch-ports
16952024-09-24T04:33:58.679ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54246 req_id = 993e4039-8107-4e9c-8cec-82d40f1ef32e response_code = 200 uri = /network-bootstore-config
16962024-09-24T04:34:04.881ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36965
16972024-09-24T04:34:04.881ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36965 req_id = 4e4f9b43-42fc-42ef-b164-5d07a9aa3c7a response_code = 200 uri = /baseboard
16982024-09-24T04:34:27.557ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54140
16992024-09-24T04:34:27.558ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54140 req_id = ce41a6eb-ee4f-49c1-8bcd-f8cc441d7a49 response_code = 200 uri = /vpc-routes
17002024-09-24T04:34:27.568ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 699 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54140 req_id = 85c09f54-78ba-4852-9ad1-3cbbb56b3334 response_code = 204 uri = /vpc-routes
17012024-09-24T04:34:27.787ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50765
17022024-09-24T04:34:27.787ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 156 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50765 req_id = c95c9884-bd08-48af-b4df-c55142f92b04 response_code = 200 uri = /v2p
17032024-09-24T04:34:27.845ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51433
17042024-09-24T04:34:27.845ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17052024-09-24T04:34:27.883ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17062024-09-24T04:34:27.938ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93269 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51433 req_id = 88e65324-4c09-4528-8fa7-42ad04a88a6d response_code = 204 uri = /switch-ports
17072024-09-24T04:34:27.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51433 req_id = 9cce2951-8678-41ef-8df8-713b8b12590f response_code = 200 uri = /network-bootstore-config
17082024-09-24T04:34:28.464ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54707
17092024-09-24T04:34:28.464ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54707 req_id = e9306638-d6e0-4e58-9f5f-cce34e830418 response_code = 200 uri = /vpc-routes
17102024-09-24T04:34:28.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54707 req_id = 17a94d69-6fdd-48ce-8f9d-44d7b967d8b7 response_code = 204 uri = /vpc-routes
17112024-09-24T04:34:28.522ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42317
17122024-09-24T04:34:28.522ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42317 req_id = 6ba9645b-962d-4335-9efe-591bea7bde3f response_code = 200 uri = /vpc-routes
17132024-09-24T04:34:28.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1023 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:42317 req_id = 9083cb40-8dfd-47c4-b5c2-b152dfcc3b86 response_code = 204 uri = /vpc-routes
17142024-09-24T04:34:28.684ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44573
17152024-09-24T04:34:28.685ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44573 req_id = 312931ac-1daa-4b2d-9d79-f46a785d4357 response_code = 200 uri = /v2p
17162024-09-24T04:34:28.705ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37019
17172024-09-24T04:34:28.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37019 req_id = 70a4b308-33ad-4e32-b956-1896a2fd34b6 response_code = 200 uri = /v2p
17182024-09-24T04:34:28.755ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57815
17192024-09-24T04:34:28.755ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17202024-09-24T04:34:28.794ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17212024-09-24T04:34:28.860ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105474 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57815 req_id = 8e9e7346-36db-4f3a-9c9d-17219f3f4d72 response_code = 204 uri = /switch-ports
17222024-09-24T04:34:28.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 860 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57815 req_id = 5c4a0f21-9a03-4989-ad0a-0f11002d1bda response_code = 200 uri = /network-bootstore-config
17232024-09-24T04:34:28.874ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42098
17242024-09-24T04:34:28.874ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17252024-09-24T04:34:28.911ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17262024-09-24T04:34:28.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93359 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42098 req_id = b2f74501-c0fc-4ae8-94f7-2cea8fc02085 response_code = 204 uri = /switch-ports
17272024-09-24T04:34:28.968ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42098 req_id = 1f99c52e-fcc9-49e2-82de-ea45da0083a5 response_code = 200 uri = /network-bootstore-config
17282024-09-24T04:34:34.915ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64190
17292024-09-24T04:34:34.915ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64190 req_id = 10435d85-c119-4009-b724-989140572b82 response_code = 200 uri = /baseboard
17302024-09-24T04:34:39.809ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:46023 remote_addr = [fd00:1122:3344:101::d]:44809 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
17312024-09-24T04:34:39.839ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 29180 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = d311bc86-fc61-4cd4-87c6-71ac5e4ad409 response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
17322024-09-24T04:34:54.387ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
17332024-09-24T04:34:57.551ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54151
17342024-09-24T04:34:57.551ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54151 req_id = 00f5420a-b76a-4a09-af41-fd9741b85002 response_code = 200 uri = /vpc-routes
17352024-09-24T04:34:57.561ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54151 req_id = b6365fcc-5e06-42af-b8e6-aea62271e59a response_code = 204 uri = /vpc-routes
17362024-09-24T04:34:57.745ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55820
17372024-09-24T04:34:57.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 944 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55820 req_id = f9ecdb7d-e237-4afe-a2be-ba9e414840e9 response_code = 200 uri = /v2p
17382024-09-24T04:34:57.800ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44751
17392024-09-24T04:34:57.800ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17402024-09-24T04:34:57.839ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17412024-09-24T04:34:57.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95296 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44751 req_id = fca89875-4eb3-4fff-aaea-c8f052e35c65 response_code = 204 uri = /switch-ports
17422024-09-24T04:34:57.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44751 req_id = c0dc35e7-8368-4b21-9006-18fb7e418f42 response_code = 200 uri = /network-bootstore-config
17432024-09-24T04:34:58.450ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47750
17442024-09-24T04:34:58.450ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47750 req_id = ea27842f-f0df-4508-a7c7-ea4fbdb70e5e response_code = 200 uri = /vpc-routes
17452024-09-24T04:34:58.465ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47750 req_id = b52a4c84-5901-4841-a5dd-afe286bbba1c response_code = 204 uri = /vpc-routes
17462024-09-24T04:34:58.473ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57636
17472024-09-24T04:34:58.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 765 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57636 req_id = 59d24707-e864-4ae4-921a-c502277b74ba response_code = 200 uri = /v2p
17482024-09-24T04:34:58.517ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53153
17492024-09-24T04:34:58.517ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 242 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53153 req_id = 1c6974c7-83c8-46d2-91b2-5e32326cab31 response_code = 200 uri = /vpc-routes
17502024-09-24T04:34:58.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53153 req_id = 4b2f220b-ac5a-4520-93f7-2df891e0a1d1 response_code = 204 uri = /vpc-routes
17512024-09-24T04:34:58.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35879
17522024-09-24T04:34:58.533ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17532024-09-24T04:34:58.577ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17542024-09-24T04:34:58.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109442 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35879 req_id = 182280f1-7236-46c6-a176-1ae2dda1777d response_code = 204 uri = /switch-ports
17552024-09-24T04:34:58.642ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53035
17562024-09-24T04:34:58.642ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46766
17572024-09-24T04:34:58.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 223 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53035 req_id = 4d799685-af59-4ccc-b354-479da0d4736d response_code = 200 uri = /v2p
17582024-09-24T04:34:58.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35879 req_id = 9af5f842-a091-47ff-af4e-a468f745761a response_code = 200 uri = /network-bootstore-config
17592024-09-24T04:34:58.644ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17602024-09-24T04:34:58.680ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17612024-09-24T04:34:58.733ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90586 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46766 req_id = 3fb8e0cc-b088-4e32-a8de-60e009c788e4 response_code = 204 uri = /switch-ports
17622024-09-24T04:34:58.734ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46766 req_id = d49139d9-8b8e-4a2b-92e5-8f910afa4acd response_code = 200 uri = /network-bootstore-config
17632024-09-24T04:35:04.948ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48955
17642024-09-24T04:35:04.948ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48955 req_id = d4548139-1294-4023-8aa5-a476bd113bc9 response_code = 200 uri = /baseboard
17652024-09-24T04:35:09.818ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9051 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = a7e5b1f2-8348-458c-a171-f88635ef763c response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
17662024-09-24T04:35:27.632ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46292
17672024-09-24T04:35:27.632ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46292 req_id = 8dcbf550-3e5d-4865-ae47-1506fef08a86 response_code = 200 uri = /vpc-routes
17682024-09-24T04:35:27.651ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46292 req_id = 88743eb0-ebaa-416b-a90b-5f4136b17ee0 response_code = 204 uri = /vpc-routes
17692024-09-24T04:35:27.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54543
17702024-09-24T04:35:27.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 163 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54543 req_id = de9ff62c-a241-4204-bede-fbbf948ee13c response_code = 200 uri = /v2p
17712024-09-24T04:35:27.884ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32926
17722024-09-24T04:35:27.884ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17732024-09-24T04:35:27.927ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17742024-09-24T04:35:27.991ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106878 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:32926 req_id = d599a433-dfd3-4b73-9e62-0b9a41902a65 response_code = 204 uri = /switch-ports
17752024-09-24T04:35:27.991ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:32926 req_id = 16f8bfdb-73f9-4655-9e60-89ae70e264ca response_code = 200 uri = /network-bootstore-config
17762024-09-24T04:35:28.557ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44754
17772024-09-24T04:35:28.558ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 883 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44754 req_id = e8014a6e-1023-4a34-94fd-2ce43a294ebd response_code = 200 uri = /vpc-routes
17782024-09-24T04:35:28.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44754 req_id = 56e99e93-7df1-461f-8510-1a8b23e6c393 response_code = 204 uri = /vpc-routes
17792024-09-24T04:35:28.636ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40145
17802024-09-24T04:35:28.636ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 362 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40145 req_id = 064bc7d2-5266-43c7-9a4a-8c6592604e06 response_code = 200 uri = /vpc-routes
17812024-09-24T04:35:28.646ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40145 req_id = 5b892c91-f77b-40bc-9750-8cabf3976045 response_code = 204 uri = /vpc-routes
17822024-09-24T04:35:28.688ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34493
17832024-09-24T04:35:28.688ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34493 req_id = 26793156-e455-47ed-af61-680eef39fd96 response_code = 200 uri = /v2p
17842024-09-24T04:35:28.722ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61569
17852024-09-24T04:35:28.722ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17862024-09-24T04:35:28.760ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17872024-09-24T04:35:28.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88742 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61569 req_id = 9c52c131-316f-4141-9335-dd406796a534 response_code = 204 uri = /switch-ports
17882024-09-24T04:35:28.811ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61062
17892024-09-24T04:35:28.811ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62617
17902024-09-24T04:35:28.811ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17912024-09-24T04:35:28.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3426 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61062 req_id = 689725fa-536a-4788-b7bc-b2874db6df24 response_code = 200 uri = /v2p
17922024-09-24T04:35:28.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3654 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61569 req_id = 2b2c7d20-2b23-4069-a5f4-7f6623ff8868 response_code = 200 uri = /network-bootstore-config
17932024-09-24T04:35:28.849ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17942024-09-24T04:35:28.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98344 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62617 req_id = ad306552-1b71-4c20-b1a3-6e27878796b6 response_code = 204 uri = /switch-ports
17952024-09-24T04:35:28.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62617 req_id = a1095a4e-d84e-4547-9f0f-648c7b12de3b response_code = 200 uri = /network-bootstore-config
17962024-09-24T04:35:34.982ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:59959
17972024-09-24T04:35:34.982ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 30 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:59959 req_id = 163adef7-b4c2-4023-aeb6-3eda58af77cf response_code = 200 uri = /baseboard
17982024-09-24T04:35:39.818ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8359 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = 8579a639-e679-4795-8d8b-335f81d4373e response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
17992024-09-24T04:35:57.576ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53627
18002024-09-24T04:35:57.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53627 req_id = 7bd4be0b-263b-4037-999c-2c3eff138e46 response_code = 200 uri = /vpc-routes
18012024-09-24T04:35:57.587ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53627 req_id = 659e32de-8344-4be1-b0d0-8554eae00211 response_code = 204 uri = /vpc-routes
18022024-09-24T04:35:57.678ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63310
18032024-09-24T04:35:57.678ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18042024-09-24T04:35:57.717ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18052024-09-24T04:35:57.774ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96087 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63310 req_id = d6eee277-3d08-450f-ae61-b1a1f8fe6ee5 response_code = 204 uri = /switch-ports
18062024-09-24T04:35:57.774ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60295
18072024-09-24T04:35:57.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 801 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60295 req_id = 88b18c94-259d-447b-95f7-2e1ccf2ad757 response_code = 200 uri = /v2p
18082024-09-24T04:35:57.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 775 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63310 req_id = 8bd6eb06-9693-4107-aa9d-53f90b0e76b5 response_code = 200 uri = /network-bootstore-config
18092024-09-24T04:35:58.426ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41930
18102024-09-24T04:35:58.427ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 801 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41930 req_id = 57887c8b-2dac-4214-849d-94e6069eea60 response_code = 200 uri = /vpc-routes
18112024-09-24T04:35:58.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41930 req_id = 71cad9bf-2db8-4aee-b520-304e60b88274 response_code = 204 uri = /vpc-routes
18122024-09-24T04:35:58.488ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57614
18132024-09-24T04:35:58.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57614 req_id = 132d4845-0195-40f7-ad9b-0fda6763e2c4 response_code = 200 uri = /vpc-routes
18142024-09-24T04:35:58.505ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51751
18152024-09-24T04:35:58.505ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51751 req_id = f64d3149-1765-45be-8133-a0b282e6dc6d response_code = 200 uri = /v2p
18162024-09-24T04:35:58.510ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57614 req_id = 04783b49-e20f-4ab0-bfc2-47670115223b response_code = 204 uri = /vpc-routes
18172024-09-24T04:35:58.522ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58596
18182024-09-24T04:35:58.522ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18192024-09-24T04:35:58.562ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18202024-09-24T04:35:58.623ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100821 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58596 req_id = df364689-9d67-4dc5-a94a-08c7019de250 response_code = 204 uri = /switch-ports
18212024-09-24T04:35:58.623ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36489
18222024-09-24T04:35:58.623ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18232024-09-24T04:35:58.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3978 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58596 req_id = c803fab6-ebca-4752-82ba-1562d95a891b response_code = 200 uri = /network-bootstore-config
18242024-09-24T04:35:58.635ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53529
18252024-09-24T04:35:58.635ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 197 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53529 req_id = 11305b6c-e9a7-4891-8f37-285ab7062413 response_code = 200 uri = /v2p
18262024-09-24T04:35:58.659ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18272024-09-24T04:35:58.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92753 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36489 req_id = 50a08a77-f508-4e8d-b8aa-480399c87993 response_code = 204 uri = /switch-ports
18282024-09-24T04:35:58.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36489 req_id = 99b9af40-05ac-483b-abed-bf8cade93111 response_code = 200 uri = /network-bootstore-config
18292024-09-24T04:36:05.015ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:32951
18302024-09-24T04:36:05.016ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 788 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:32951 req_id = 660cc7b9-5b3c-4de8-8df1-e0e638f15e23 response_code = 200 uri = /baseboard
18312024-09-24T04:36:09.817ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7804 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = ca9ef6f6-e994-4a95-8343-e6047b1721ca response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
18322024-09-24T04:36:27.627ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38866
18332024-09-24T04:36:27.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 883 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38866 req_id = 5673267d-6352-48d7-9b6e-ba2aa0754da6 response_code = 200 uri = /vpc-routes
18342024-09-24T04:36:27.652ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2264 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38866 req_id = 85ea15d1-76ff-44be-93b3-295992af7f83 response_code = 204 uri = /vpc-routes
18352024-09-24T04:36:27.836ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51748
18362024-09-24T04:36:27.836ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51748 req_id = 641d595f-fea6-40f6-ab8b-b47dc3b718cf response_code = 200 uri = /v2p
18372024-09-24T04:36:27.880ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36319
18382024-09-24T04:36:27.880ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18392024-09-24T04:36:27.920ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18402024-09-24T04:36:27.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102562 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36319 req_id = 064b0e2e-2751-45ca-a932-9d024ae2aab2 response_code = 204 uri = /switch-ports
18412024-09-24T04:36:27.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36319 req_id = fc39eb3b-0a81-4b79-9ef1-03ce6aa338b5 response_code = 200 uri = /network-bootstore-config
18422024-09-24T04:36:28.477ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40473
18432024-09-24T04:36:28.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40473 req_id = 5bfb520e-b945-4c7f-8fc0-0b230de73929 response_code = 200 uri = /vpc-routes
18442024-09-24T04:36:28.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 969 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40473 req_id = aa61fdc7-bb70-4ca6-be6a-ce1ee97fa197 response_code = 204 uri = /vpc-routes
18452024-09-24T04:36:28.572ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44433
18462024-09-24T04:36:28.572ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44433 req_id = 5fc5fa0d-2f8b-4340-91fa-851a7511fb6c response_code = 200 uri = /vpc-routes
18472024-09-24T04:36:28.583ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44433 req_id = e0e0cb94-8b28-4281-90d0-b217dc691637 response_code = 204 uri = /vpc-routes
18482024-09-24T04:36:28.632ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64960
18492024-09-24T04:36:28.632ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64960 req_id = 899c630f-2dde-4e99-8fab-cb416f2f38e6 response_code = 200 uri = /v2p
18502024-09-24T04:36:28.671ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40232
18512024-09-24T04:36:28.671ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18522024-09-24T04:36:28.709ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18532024-09-24T04:36:28.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96113 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40232 req_id = 0d56369e-43de-43fe-a60d-0abab24c879b response_code = 204 uri = /switch-ports
18542024-09-24T04:36:28.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46224
18552024-09-24T04:36:28.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57553
18562024-09-24T04:36:28.767ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18572024-09-24T04:36:28.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 388 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40232 req_id = c1f65f91-96e2-475e-8c1d-ce914da39730 response_code = 200 uri = /network-bootstore-config
18582024-09-24T04:36:28.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9455 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57553 req_id = d532f7b5-a229-4879-9772-528ff3f26368 response_code = 200 uri = /v2p
18592024-09-24T04:36:28.808ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18602024-09-24T04:36:28.863ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96008 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46224 req_id = f38040c8-0ed8-4891-930c-aee54588a30f response_code = 204 uri = /switch-ports
18612024-09-24T04:36:28.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46224 req_id = b129c06c-17cd-453f-86d1-27f3b55362d6 response_code = 200 uri = /network-bootstore-config
18622024-09-24T04:36:35.049ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:43037
18632024-09-24T04:36:35.049ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 36 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:43037 req_id = 719c954d-4398-479b-9ba9-ffb3c5accb85 response_code = 200 uri = /baseboard
18642024-09-24T04:36:39.824ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5375 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = 65ae102e-6d1c-4b83-8916-a755faf6b047 response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
18652024-09-24T04:36:57.572ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64479
18662024-09-24T04:36:57.572ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64479 req_id = 3cada193-a549-4f73-9215-5a30e074f080 response_code = 200 uri = /vpc-routes
18672024-09-24T04:36:57.583ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 653 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64479 req_id = a514c8c4-be7d-4665-8caa-e2daf7b4b6c1 response_code = 204 uri = /vpc-routes
18682024-09-24T04:36:57.674ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61239
18692024-09-24T04:36:57.674ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61239 req_id = 6dc81a63-3649-4739-baaf-253bbec66887 response_code = 200 uri = /v2p
18702024-09-24T04:36:57.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63090
18712024-09-24T04:36:57.687ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18722024-09-24T04:36:57.728ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18732024-09-24T04:36:57.785ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98651 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63090 req_id = 6eb20590-93f2-44c6-b5b8-e7df4b70ed3a response_code = 204 uri = /switch-ports
18742024-09-24T04:36:57.786ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63090 req_id = d51f846d-779e-42e5-8aac-e15d3b981116 response_code = 200 uri = /network-bootstore-config
18752024-09-24T04:36:58.364ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54827
18762024-09-24T04:36:58.364ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54827 req_id = 1335fcb0-3684-4f52-b4fb-3aa09dfc1cf4 response_code = 200 uri = /vpc-routes
18772024-09-24T04:36:58.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54827 req_id = e4510198-4bf6-427e-bc1f-34de13c17f80 response_code = 204 uri = /vpc-routes
18782024-09-24T04:36:58.396ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46961
18792024-09-24T04:36:58.396ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46961 req_id = fcbdbc09-cf9d-4491-9107-b58d58c170d7 response_code = 200 uri = /vpc-routes
18802024-09-24T04:36:58.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46961 req_id = b3904e9d-24b4-4b8e-9d04-36449f20ed94 response_code = 204 uri = /vpc-routes
18812024-09-24T04:36:58.467ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58017
18822024-09-24T04:36:58.468ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18832024-09-24T04:36:58.507ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18842024-09-24T04:36:58.565ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97876 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58017 req_id = 4e03c515-1d32-4447-a48f-a78cef99b9a0 response_code = 204 uri = /switch-ports
18852024-09-24T04:36:58.566ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63832
18862024-09-24T04:36:58.566ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49448
18872024-09-24T04:36:58.566ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44866
18882024-09-24T04:36:58.567ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18892024-09-24T04:36:58.567ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 423 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58017 req_id = 2b84e03f-feee-441f-af78-eec01b6a7a0e response_code = 200 uri = /network-bootstore-config
18902024-09-24T04:36:58.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5527 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44866 req_id = 42fb4bef-9a6b-4ac6-a368-ae2ab988d2a1 response_code = 200 uri = /v2p
18912024-09-24T04:36:58.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5479 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49448 req_id = 9fcabbf8-d30e-415e-98b2-6ee156496871 response_code = 200 uri = /v2p
18922024-09-24T04:36:58.607ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18932024-09-24T04:36:58.662ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95818 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63832 req_id = 0e186d46-e1c0-4a1d-8a05-c55224d2f8f8 response_code = 204 uri = /switch-ports
18942024-09-24T04:36:58.662ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63832 req_id = 6380d6d0-ecba-4149-ad90-0be85600fc5e response_code = 200 uri = /network-bootstore-config
18952024-09-24T04:37:05.100ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:49312
18962024-09-24T04:37:05.101ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 674 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:49312 req_id = c90919d7-f255-4c74-a7fb-a24b0f652046 response_code = 200 uri = /baseboard
18972024-09-24T04:37:09.866ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6413 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = 871791b6-cb54-4d99-ad53-6e634ae9ae12 response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
18982024-09-24T04:37:27.712ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35926
18992024-09-24T04:37:27.712ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35926 req_id = e1844126-b202-4526-a632-4478968b058a response_code = 200 uri = /vpc-routes
19002024-09-24T04:37:27.726ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 852 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35926 req_id = fda2186b-9b97-44b1-bf93-8aba52c3436f response_code = 204 uri = /vpc-routes
19012024-09-24T04:37:27.856ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62750
19022024-09-24T04:37:27.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62750 req_id = af2b80f1-9307-410f-8cf9-9003121ccba9 response_code = 200 uri = /v2p
19032024-09-24T04:37:27.973ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52728
19042024-09-24T04:37:27.973ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19052024-09-24T04:37:28.014ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19062024-09-24T04:37:28.072ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99065 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52728 req_id = 41f21b5d-87c0-41d7-8575-b825c875bb35 response_code = 204 uri = /switch-ports
19072024-09-24T04:37:28.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 741 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52728 req_id = 26c9280d-2038-4130-a9bd-f21322cfd196 response_code = 200 uri = /network-bootstore-config
19082024-09-24T04:37:28.538ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58407
19092024-09-24T04:37:28.538ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58407 req_id = 4f7dd7ea-11c0-466a-8cb3-81f05e1e7822 response_code = 200 uri = /vpc-routes
19102024-09-24T04:37:28.558ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58407 req_id = a101be59-4e37-40fe-bbe7-3a5b30215f6c response_code = 204 uri = /vpc-routes
19112024-09-24T04:37:28.581ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33731
19122024-09-24T04:37:28.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33731 req_id = 8e34d7bf-2204-45d9-93eb-3ed46668c213 response_code = 200 uri = /vpc-routes
19132024-09-24T04:37:28.592ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33731 req_id = f583ee0a-9c85-4c23-9117-cb90a5f82904 response_code = 204 uri = /vpc-routes
19142024-09-24T04:37:28.753ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51406
19152024-09-24T04:37:28.753ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 737 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51406 req_id = 64383acc-2052-47fa-8fa5-726aedb5c45e response_code = 200 uri = /v2p
19162024-09-24T04:37:28.761ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45645
19172024-09-24T04:37:28.761ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45645 req_id = 5ef533d4-4e4b-4f19-8993-a410c68f698a response_code = 200 uri = /v2p
19182024-09-24T04:37:28.766ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47073
19192024-09-24T04:37:28.767ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19202024-09-24T04:37:28.810ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19212024-09-24T04:37:28.869ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102598 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47073 req_id = 6f7e68ce-a644-4051-945e-91471c1029f7 response_code = 204 uri = /switch-ports
19222024-09-24T04:37:28.869ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49502
19232024-09-24T04:37:28.869ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19242024-09-24T04:37:28.904ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19252024-09-24T04:37:28.960ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90488 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49502 req_id = 245b1bf1-61b8-4c92-b524-bbd22035d5f4 response_code = 204 uri = /switch-ports
19262024-09-24T04:37:28.960ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47073 req_id = e15bd469-cbe6-41ff-82df-8300a1ff2c19 response_code = 200 uri = /network-bootstore-config
19272024-09-24T04:37:28.960ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49502 req_id = e71206a6-ee49-4195-9580-d2e36add109c response_code = 200 uri = /network-bootstore-config
19282024-09-24T04:37:35.136ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53656
19292024-09-24T04:37:35.136ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53656 req_id = 480d7963-4cf5-46e8-8860-f724ff543109 response_code = 200 uri = /baseboard
19302024-09-24T04:37:39.867ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7599 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = f8fdd3dd-7ea1-4d11-8fac-6a669d47cb2b response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
19312024-09-24T04:37:57.508ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64794
19322024-09-24T04:37:57.509ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 988 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64794 req_id = 589fa1c7-278e-445f-8cd3-87d68e40e83b response_code = 200 uri = /vpc-routes
19332024-09-24T04:37:57.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 885 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64794 req_id = 4629721f-809e-4617-afc5-b9f102ecfd06 response_code = 204 uri = /vpc-routes
19342024-09-24T04:37:57.714ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34782
19352024-09-24T04:37:57.714ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34782 req_id = 6692533f-d408-4ab6-a691-36256ae77e8b response_code = 200 uri = /v2p
19362024-09-24T04:37:57.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34951
19372024-09-24T04:37:57.767ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19382024-09-24T04:37:57.806ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19392024-09-24T04:37:57.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97835 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34951 req_id = 8549f296-8db5-4442-9727-981f0277fd59 response_code = 204 uri = /switch-ports
19402024-09-24T04:37:57.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34951 req_id = d86a04d7-da29-4197-b098-8b6e02efc748 response_code = 200 uri = /network-bootstore-config
19412024-09-24T04:37:58.335ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:32843
19422024-09-24T04:37:58.336ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:32843 req_id = c1ee6568-86d5-4de5-9004-a760bd887e83 response_code = 200 uri = /vpc-routes
19432024-09-24T04:37:58.417ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37080
19442024-09-24T04:37:58.418ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 180 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37080 req_id = f974ce92-1a9b-4726-9eaa-7f9d41f0329a response_code = 200 uri = /v2p
19452024-09-24T04:37:58.420ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:32843 req_id = d5b1efcf-890d-46af-bb23-610197d2a6ef response_code = 204 uri = /vpc-routes
19462024-09-24T04:37:58.449ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41163
19472024-09-24T04:37:58.450ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 760 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41163 req_id = e362c861-d487-4138-9228-b705f405d502 response_code = 200 uri = /vpc-routes
19482024-09-24T04:37:58.459ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41163 req_id = 8e84a575-422c-4c6a-b50b-ee30ff352da9 response_code = 204 uri = /vpc-routes
19492024-09-24T04:37:58.475ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50730
19502024-09-24T04:37:58.476ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19512024-09-24T04:37:58.514ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19522024-09-24T04:37:58.573ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98071 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50730 req_id = 6ebe8a3e-0878-4795-9e3c-12e37ba7eaf3 response_code = 204 uri = /switch-ports
19532024-09-24T04:37:58.573ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40475
19542024-09-24T04:37:58.574ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54843
19552024-09-24T04:37:58.574ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50730 req_id = d097d552-bc99-4a99-9efc-a490fe8892ad response_code = 200 uri = /network-bootstore-config
19562024-09-24T04:37:58.574ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19572024-09-24T04:37:58.575ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 921 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40475 req_id = e881f84f-a6cd-4a59-878a-19479acd173f response_code = 200 uri = /v2p
19582024-09-24T04:37:58.611ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19592024-09-24T04:37:58.666ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92258 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54843 req_id = 0a68c63e-9c92-4c20-b524-d4b1b69e4022 response_code = 204 uri = /switch-ports
19602024-09-24T04:37:58.666ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54843 req_id = 3bd72f2c-1fa6-43e9-bae8-c1a7bba4bd1d response_code = 200 uri = /network-bootstore-config
19612024-09-24T04:38:05.169ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35138
19622024-09-24T04:38:05.169ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35138 req_id = fb1f4f40-f816-4517-94c7-f2ecce5d8cef response_code = 200 uri = /baseboard
19632024-09-24T04:38:09.870ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 10330 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = 540443a2-e859-489b-8ea0-f40705d6683d response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
19642024-09-24T04:38:27.629ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37978
19652024-09-24T04:38:27.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37978 req_id = ab7f16b4-7c5c-4d71-acbe-e4bc5ffcf2c2 response_code = 200 uri = /vpc-routes
19662024-09-24T04:38:27.786ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34582
19672024-09-24T04:38:27.786ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 156 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34582 req_id = 1faca0f3-c325-4ca7-b283-28729e863b72 response_code = 200 uri = /v2p
19682024-09-24T04:38:27.793ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 796 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37978 req_id = dcbb4e5f-b353-4d16-bbd7-38dba7f0856d response_code = 204 uri = /vpc-routes
19692024-09-24T04:38:27.919ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62432
19702024-09-24T04:38:27.919ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
19712024-09-24T04:38:27.919ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19722024-09-24T04:38:27.919ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19732024-09-24T04:38:27.919ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19742024-09-24T04:38:27.919ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
19752024-09-24T04:38:27.919ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19762024-09-24T04:38:27.919ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19772024-09-24T04:38:27.919ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 292 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62432 req_id = a7eedc75-7dc3-4ddd-a609-f286817e7484 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
19782024-09-24T04:38:27.973ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58611
19792024-09-24T04:38:27.973ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19802024-09-24T04:38:28.013ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19812024-09-24T04:38:28.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99718 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58611 req_id = 23b74856-f72b-4751-bed3-bc8f70b1061c response_code = 204 uri = /switch-ports
19822024-09-24T04:38:28.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58611 req_id = 87cdcc3f-96bf-4dae-8b5d-d732f6c522dd response_code = 200 uri = /network-bootstore-config
19832024-09-24T04:38:28.589ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37922
19842024-09-24T04:38:28.589ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 197 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37922 req_id = 0bd22354-bd24-4d98-8428-2bd5999b33cd response_code = 200 uri = /v2p
19852024-09-24T04:38:28.605ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39004
19862024-09-24T04:38:28.605ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39004 req_id = 00a04361-3752-4cee-9a25-cacbde7285db response_code = 200 uri = /vpc-routes
19872024-09-24T04:38:28.616ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39004 req_id = d9c7af28-5236-4950-a49e-d0fef6a3bd88 response_code = 204 uri = /vpc-routes
19882024-09-24T04:38:28.671ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40346
19892024-09-24T04:38:28.671ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40346 req_id = c5c6d0b2-8647-4739-a5ff-cf4b8df2da76 response_code = 200 uri = /vpc-routes
19902024-09-24T04:38:28.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40346 req_id = 7b6b2e04-29c3-458f-a6b0-e6823a311d24 response_code = 204 uri = /vpc-routes
19912024-09-24T04:38:28.712ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59838
19922024-09-24T04:38:28.712ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59838 req_id = 3345946a-f30a-4e2d-8407-fda55a41375e response_code = 200 uri = /v2p
19932024-09-24T04:38:28.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62220
19942024-09-24T04:38:28.758ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19952024-09-24T04:38:28.799ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
19962024-09-24T04:38:28.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102934 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62220 req_id = 0be153bc-c853-40d2-bc2f-391e85e5729f response_code = 204 uri = /switch-ports
19972024-09-24T04:38:28.861ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34628
19982024-09-24T04:38:28.862ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
19992024-09-24T04:38:28.896ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
20002024-09-24T04:38:28.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89723 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34628 req_id = 24560cc6-430e-4405-9409-f47663c5d050 response_code = 204 uri = /switch-ports
20012024-09-24T04:38:28.952ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33380
20022024-09-24T04:38:28.952ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62220 req_id = 3e3f8cd6-d1f9-4a31-bc99-e92c3a5008e1 response_code = 200 uri = /network-bootstore-config
20032024-09-24T04:38:28.952ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37657
20042024-09-24T04:38:28.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 987 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34628 req_id = c48c8dd9-6074-49ed-a884-7d15e6dfd4f2 response_code = 200 uri = /network-bootstore-config
20052024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
20062024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 20f40ca3-cd4c-4b33-94d2-e8c3ba02484f, kind: Service { id: eb18c719-9991-4594-9ed0-1ee13a881361 }, name: Name("external-dns-eb18c719-9991-4594-9ed0-1ee13a881361"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f7aebe9f-1122-4300-a2e8-f053b218c332, kind: Service { id: 3a3c9bd7-6179-4647-88be-743dbba6ce2f }, name: Name("external-dns-3a3c9bd7-6179-4647-88be-743dbba6ce2f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 91])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60c46c7d-3b44-42db-8cd2-2720179b0888, kind: Service { id: 9c4a6c7c-d0f3-4040-893d-c86718cd5863 }, name: Name("nexus-9c4a6c7c-d0f3-4040-893d-c86718cd5863"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 233])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 979cc46d-7722-45c4-9e68-e64e9dd80de0, kind: Service { id: 28b4d493-d618-4550-b832-f5a8700dda16 }, name: Name("nexus-28b4d493-d618-4550-b832-f5a8700dda16"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 24])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dd24fed4-200c-4d1b-882d-d71fc6f9ddd2, kind: Service { id: 19d1b3bf-4d38-489c-a831-139faa2845a8 }, name: Name("nexus-19d1b3bf-4d38-489c-a831-139faa2845a8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 131])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
20072024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20082024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20092024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20102024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
20112024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20122024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20132024-09-24T04:38:28.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1058 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37657 req_id = f5804414-6111-49fc-a0f2-9e0486538bc7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
20142024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20152024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20162024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20172024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
20182024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20192024-09-24T04:38:28.953ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20202024-09-24T04:38:28.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1253 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33380 req_id = ac2fb6d8-e706-494d-9116-c93ed87ddeb7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
20212024-09-24T04:38:35.206ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44557
20222024-09-24T04:38:35.206ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44557 req_id = 7db918d1-cb89-4db5-8697-cf71702b1efd response_code = 200 uri = /baseboard
20232024-09-24T04:38:39.867ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7270 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = 0765e644-62cc-487f-adf7-b536a5aa7323 response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
20242024-09-24T04:38:57.681ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44026
20252024-09-24T04:38:57.682ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44026 req_id = 12e47c45-25e6-46df-8eaa-0ebfe052f469 response_code = 200 uri = /vpc-routes
20262024-09-24T04:38:57.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44026 req_id = 4c98eba4-a761-4801-ab15-d2d51b7a07df response_code = 204 uri = /vpc-routes
20272024-09-24T04:38:57.746ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43909
20282024-09-24T04:38:57.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43909 req_id = f3ac6581-9b9c-425d-a310-353cf6e691a2 response_code = 200 uri = /v2p
20292024-09-24T04:38:57.757ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50518
20302024-09-24T04:38:57.757ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
20312024-09-24T04:38:57.799ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
20322024-09-24T04:38:57.855ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97454 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50518 req_id = d31e3c3b-cf8b-4227-a942-b9c528131759 response_code = 204 uri = /switch-ports
20332024-09-24T04:38:57.855ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50518 req_id = bec28456-c45c-4063-9cfd-ebeb6d70e675 response_code = 200 uri = /network-bootstore-config
20342024-09-24T04:38:58.456ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46048
20352024-09-24T04:38:58.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46048 req_id = 2f3225e5-c935-4aa7-8220-d0765e2e4e49 response_code = 200 uri = /vpc-routes
20362024-09-24T04:38:58.465ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46048 req_id = b605f3e4-1b5e-4e85-abcb-ccd9ddff31c7 response_code = 204 uri = /vpc-routes
20372024-09-24T04:38:58.465ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38967
20382024-09-24T04:38:58.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 715 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38967 req_id = c0ea2a09-ee17-4e9e-a69f-e774e3426596 response_code = 200 uri = /vpc-routes
20392024-09-24T04:38:58.477ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38967 req_id = c425a6ea-98d3-45ab-92cb-8286ac07e6af response_code = 204 uri = /vpc-routes
20402024-09-24T04:38:58.544ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64893
20412024-09-24T04:38:58.544ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
20422024-09-24T04:38:58.587ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
20432024-09-24T04:38:58.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102830 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64893 req_id = 10a7e71b-3120-438f-829d-7a043bac8f8b response_code = 204 uri = /switch-ports
20442024-09-24T04:38:58.647ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50684
20452024-09-24T04:38:58.647ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41741
20462024-09-24T04:38:58.647ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35642
20472024-09-24T04:38:58.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 228 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64893 req_id = 325b212d-7331-447a-b4b4-1be210fdd860 response_code = 200 uri = /network-bootstore-config
20482024-09-24T04:38:58.648ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
20492024-09-24T04:38:58.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 974 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50684 req_id = f19ad388-1d75-48f1-a26a-fef324e1f3f6 response_code = 200 uri = /v2p
20502024-09-24T04:38:58.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1016 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35642 req_id = ad688ed9-9bc8-4da0-a8da-ded8bed9ce2b response_code = 200 uri = /v2p
20512024-09-24T04:38:58.688ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
20522024-09-24T04:38:58.744ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97276 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:41741 req_id = 8ebfd57e-eb44-46da-8521-8f33372279a3 response_code = 204 uri = /switch-ports
20532024-09-24T04:38:58.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41741 req_id = 0ce89502-54fb-472f-8422-f5852e2bd2ca response_code = 200 uri = /network-bootstore-config
20542024-09-24T04:39:05.238ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:43855
20552024-09-24T04:39:05.238ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 637 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:43855 req_id = 0c6bc41e-a3aa-4aea-aec7-acbbcb2e05d8 response_code = 200 uri = /baseboard
20562024-09-24T04:39:09.867ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8148 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = 7834c4d4-5c34-46b1-b017-340f1b600486 response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
20572024-09-24T04:39:27.748ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55063
20582024-09-24T04:39:27.748ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55063 req_id = 0914c194-4015-4fb8-a777-cafe68434802 response_code = 200 uri = /vpc-routes
20592024-09-24T04:39:27.755ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42604
20602024-09-24T04:39:27.755ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42604 req_id = f6d07952-b040-469d-9672-e9cb2defa8d6 response_code = 200 uri = /v2p
20612024-09-24T04:39:27.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 771 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55063 req_id = 652f527a-869d-49f3-85a7-cc0ab1c235fc response_code = 204 uri = /vpc-routes
20622024-09-24T04:39:28.065ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36358
20632024-09-24T04:39:28.065ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
20642024-09-24T04:39:28.104ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
20652024-09-24T04:39:28.163ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97452 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36358 req_id = 311a0644-a227-48f9-9b67-c2f987629e66 response_code = 204 uri = /switch-ports
20662024-09-24T04:39:28.164ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 652 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36358 req_id = c145476d-cfb0-4d10-8d22-3fe0eb441d9d response_code = 200 uri = /network-bootstore-config
20672024-09-24T04:39:28.588ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45564
20682024-09-24T04:39:28.589ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45564 req_id = 307bdefb-92dc-4fc0-987f-21325d172adc response_code = 200 uri = /vpc-routes
20692024-09-24T04:39:28.599ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:45564 req_id = bafdac14-ab3c-477e-81d2-b127fb8b1cc5 response_code = 204 uri = /vpc-routes
20702024-09-24T04:39:28.691ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57552
20712024-09-24T04:39:28.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57552 req_id = 890612cf-e200-49d8-9079-26e09e4402a5 response_code = 200 uri = /v2p
20722024-09-24T04:39:28.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41175
20732024-09-24T04:39:28.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41175 req_id = 5488ea4a-2798-4df8-9a03-37b261cfd1bc response_code = 200 uri = /vpc-routes
20742024-09-24T04:39:28.724ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:32887
20752024-09-24T04:39:28.724ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:32887 req_id = a18bf94b-8d12-48ee-b461-223dddce78e9 response_code = 200 uri = /v2p
20762024-09-24T04:39:28.726ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41175 req_id = 98b604c9-9071-46be-a22b-cf6585dc79b5 response_code = 204 uri = /vpc-routes
20772024-09-24T04:39:28.820ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35827
20782024-09-24T04:39:28.820ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
20792024-09-24T04:39:28.860ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
20802024-09-24T04:39:28.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95840 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35827 req_id = 2c3c6c56-fe75-47fb-abda-e3be617d7c8a response_code = 204 uri = /switch-ports
20812024-09-24T04:39:28.916ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61566
20822024-09-24T04:39:28.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 290 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35827 req_id = f6d7f153-415e-4e79-8e6e-dcf8203269e4 response_code = 200 uri = /network-bootstore-config
20832024-09-24T04:39:28.917ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
20842024-09-24T04:39:28.955ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
20852024-09-24T04:39:29.008ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91887 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61566 req_id = 562fc1c1-9e67-47b4-8ab6-647842d7248f response_code = 204 uri = /switch-ports
20862024-09-24T04:39:29.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61566 req_id = 6ee5b56f-ade4-4d38-9bb8-24df47d60926 response_code = 200 uri = /network-bootstore-config
20872024-09-24T04:39:35.271ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:43820
20882024-09-24T04:39:35.272ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:43820 req_id = 02bd0d03-6fe8-4d6a-be9f-719a028116dd response_code = 200 uri = /baseboard
20892024-09-24T04:39:39.841ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:197
20902024-09-24T04:39:39.841ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20912024-09-24T04:39:39.877ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1971, tv_nsec: 456159575 } time_to_next_cleanup = 599.99999784s
20922024-09-24T04:39:39.978ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6801 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = af2b5ab8-f57c-4d95-acdc-71ed492954f8 response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
20932024-09-24T04:39:54.388ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
20942024-09-24T04:39:57.600ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61955
20952024-09-24T04:39:57.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61955 req_id = cc90a9bf-7226-493e-b5c6-a18f023afe36 response_code = 200 uri = /vpc-routes
20962024-09-24T04:39:57.608ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61955 req_id = 99cc7fd2-018a-4f3d-915b-1058c1c77bc9 response_code = 204 uri = /vpc-routes
20972024-09-24T04:39:57.699ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52035
20982024-09-24T04:39:57.699ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
20992024-09-24T04:39:57.741ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
21002024-09-24T04:39:57.802ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102432 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52035 req_id = 26e379b4-d920-4d7a-9e48-ea6867860ae7 response_code = 204 uri = /switch-ports
21012024-09-24T04:39:57.802ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61208
21022024-09-24T04:39:57.802ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52035 req_id = dff80417-f63d-45b5-a86e-20d93380212b response_code = 200 uri = /network-bootstore-config
21032024-09-24T04:39:57.802ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 216 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61208 req_id = 7212d570-2e1f-4dc7-844a-197e6bbc6471 response_code = 200 uri = /v2p
21042024-09-24T04:39:58.368ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54698
21052024-09-24T04:39:58.368ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54698 req_id = 26c12cba-d524-472d-8374-125de322e1ce response_code = 200 uri = /v2p
21062024-09-24T04:39:58.447ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61654
21072024-09-24T04:39:58.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61654 req_id = bb385ecc-74bf-4cb5-9806-5e4ded30a558 response_code = 200 uri = /vpc-routes
21082024-09-24T04:39:58.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61654 req_id = 56ecef72-265d-4c3d-a331-892f0e7dd396 response_code = 204 uri = /vpc-routes
21092024-09-24T04:39:58.489ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49825
21102024-09-24T04:39:58.489ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49825 req_id = b28cf208-d16a-4251-9e99-8f11202a764a response_code = 200 uri = /vpc-routes
21112024-09-24T04:39:58.500ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49825 req_id = 119df965-e59f-4ab2-9b75-f780cce57654 response_code = 204 uri = /vpc-routes
21122024-09-24T04:39:58.551ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57757
21132024-09-24T04:39:58.552ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 895 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57757 req_id = 3fea9a2e-68e0-4e12-8363-54734f0777ee response_code = 200 uri = /v2p
21142024-09-24T04:39:58.619ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39013
21152024-09-24T04:39:58.619ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
21162024-09-24T04:39:58.676ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
21172024-09-24T04:39:58.748ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129058 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39013 req_id = c5f5b5a2-25b9-451b-9e9d-a1563614886d response_code = 204 uri = /switch-ports
21182024-09-24T04:39:58.748ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43480
21192024-09-24T04:39:58.748ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
21202024-09-24T04:39:58.784ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
21212024-09-24T04:39:58.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100038 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43480 req_id = 6a95991b-1a1e-491d-9028-0673453ca05d response_code = 204 uri = /switch-ports
21222024-09-24T04:39:58.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39013 req_id = 29362a8c-4c10-4fe5-b872-75e519a7bcd6 response_code = 200 uri = /network-bootstore-config
21232024-09-24T04:39:58.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43480 req_id = d929e61e-4411-4419-8feb-e40f35ac7c17 response_code = 200 uri = /network-bootstore-config
21242024-09-24T04:40:05.306ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48445
21252024-09-24T04:40:05.306ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48445 req_id = 6c1d9fc0-92e6-4737-aa1a-aab093575dad response_code = 200 uri = /baseboard
21262024-09-24T04:40:06.941ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43319
21272024-09-24T04:40:06.941ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43319 req_id = 1092ddd8-530c-413c-a078-9b4132a4ad65 response_code = 200 uri = /v2p
21282024-09-24T04:40:06.942ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(558280) }
21292024-09-24T04:40:06.942ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 923 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43319 req_id = 378d9a2b-3a54-4303-bb74-0be693c433b5 response_code = 204 uri = /v2p
21302024-09-24T04:40:07.149ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44260
21312024-09-24T04:40:07.158ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:539 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(558280))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-78870f01934", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 0441fc30-9895-4095-8777-18d88d5dd6f8, read_only_parent: Some(Volume { block_size: 512, id: 42d7b890-f16c-46e5-a8a8-f94449c5b23f, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: c50a9a61-01e7-468a-b831-e6c53ed28bef, key: Some("gXFjKgBJDtYWBa6wY28MEr+tdgG8JQcfhkoai+M26Xg="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::17]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0441fc30-9895-4095-8777-18d88d5dd6f8, key: Some("hx0g7SBhaJIALn/vOAhVJVU3paB8Tp+LP61Fhdf4ihc="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some(..) } instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc metadata = InstanceMetadata { silo_id: a434c039-2f1c-4c21-bca4-46a81dca0bd0, project_id: 05dd5cb3-e855-44c6-a248-f7c763355b17 } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-24T04:40:07.148851337Z }
21322024-09-24T04:40:07.158ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:572 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc migration_id = None propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21332024-09-24T04:40:07.159ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1008 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc migration_id = None propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(558280))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d099f902-0234-40f8-8d1c-756e0b76a24e, kind: Instance { id: 78d2b282-95f0-4b45-a7c2-e9d0874345dc }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(558280), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-78870f01934", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 0441fc30-9895-4095-8777-18d88d5dd6f8, read_only_parent: Some(Volume { block_size: 512, id: 42d7b890-f16c-46e5-a8a8-f94449c5b23f, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: c50a9a61-01e7-468a-b831-e6c53ed28bef, key: Some("gXFjKgBJDtYWBa6wY28MEr+tdgG8JQcfhkoai+M26Xg="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::17]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0441fc30-9895-4095-8777-18d88d5dd6f8, key: Some("hx0g7SBhaJIALn/vOAhVJVU3paB8Tp+LP61Fhdf4ihc="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-24T04:40:07.148851337Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
21342024-09-24T04:40:07.159ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 10054 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44260 req_id = 746e001f-7b74-4dad-a075-68521c57d5c0 response_code = 200 uri = /vmms/a0b54e19-f881-4c26-ad6c-958622064475
21352024-09-24T04:40:07.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38852
21362024-09-24T04:40:07.262ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 311 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38852 req_id = 049519a1-3398-4912-9a21-bed69def6e46 response_code = 200 uri = /vmms/a0b54e19-f881-4c26-ad6c-958622064475/state
21372024-09-24T04:40:07.262ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 558280 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
21382024-09-24T04:40:07.262ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21392024-09-24T04:40:07.303ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475 file = illumos-utils/src/zone.rs:316 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21402024-09-24T04:40:07.328ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475 file = illumos-utils/src/zone.rs:354 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21412024-09-24T04:40:08.687ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21422024-09-24T04:40:08.687ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475 zone file = illumos-utils/src/running_zone.rs:488 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21432024-09-24T04:40:08.945ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 28.130188ms file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21442024-09-24T04:40:08.983ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 116.620205ms file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21452024-09-24T04:40:09.114ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 219.790586ms file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21462024-09-24T04:40:09.346ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 318.87112ms file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21472024-09-24T04:40:09.677ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 1.143227142s file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21482024-09-24T04:40:09.980ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9000 local_addr = [fd00:1122:3344:101::1]:46023 method = GET remote_addr = [fd00:1122:3344:101::d]:44809 req_id = c80b701e-c275-4602-8868-525353bd543b response_code = 200 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67 uri = /223823d0-6342-4be2-8483-3bbd7919ea67
21492024-09-24T04:40:10.832ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 769.233364ms file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21502024-09-24T04:40:11.613ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 1.151225232s file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21512024-09-24T04:40:12.776ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475") failed: Property not found. retry in 1.43770493s file = illumos-utils/src/svc.rs:36 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 zone = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
21522024-09-24T04:40:14.243ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475 file = sled-agent/src/instance.rs:1466 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21532024-09-24T04:40:14.253ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1475 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21542024-09-24T04:40:14.293ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1499 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21552024-09-24T04:40:14.443ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:744 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21562024-09-24T04:40:14.444ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-09-24T04:40:14.444831186Z } file = sled-agent/src/instance.rs:594 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21572024-09-24T04:40:14.446ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-24T04:40:14.444831186Z } propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21582024-09-24T04:40:14.446ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-24T04:40:14.444831186Z }, migration_in: None, migration_out: None }
21592024-09-24T04:40:14.473ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-09-24T04:40:14.473611858Z } file = sled-agent/src/instance.rs:594 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21602024-09-24T04:40:14.473ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-24T04:40:14.473611858Z } propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
21612024-09-24T04:40:14.473ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-24T04:40:14.473611858Z }, migration_in: None, migration_out: None }
21622024-09-24T04:40:27.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50880
21632024-09-24T04:40:27.690ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50880 req_id = a88d616f-ec71-4e66-bf87-55e20d288696 response_code = 200 uri = /vpc-routes
21642024-09-24T04:40:27.777ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32864
21652024-09-24T04:40:27.778ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 183 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:32864 req_id = 76d8d247-ee03-490b-9a92-727c37423362 response_code = 200 uri = /vmms/a0b54e19-f881-4c26-ad6c-958622064475/state
21662024-09-24T04:40:27.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 319 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50880 req_id = 1eed19cf-10a7-4f2f-a2f1-6a7c43c0a3c4 response_code = 204 uri = /vpc-routes
21672024-09-24T04:40:27.966ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51950
21682024-09-24T04:40:27.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51950 req_id = df9b49b9-b37f-48bc-8aa5-2ca8e271bc5b response_code = 200 uri = /v2p
21692024-09-24T04:40:27.969ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57190
21702024-09-24T04:40:27.970ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
21712024-09-24T04:40:28.010ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
21722024-09-24T04:40:28.065ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95591 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57190 req_id = 922e4dd8-d320-48c4-9722-5f52dea0e5eb response_code = 204 uri = /switch-ports
21732024-09-24T04:40:28.065ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57190 req_id = 3222de7f-965a-4a69-8f33-89679f920187 response_code = 200 uri = /network-bootstore-config
21742024-09-24T04:40:28.531ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59161
21752024-09-24T04:40:28.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 900 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59161 req_id = 94c7c022-da4e-4b2f-9571-34054bd4628b response_code = 200 uri = /vmms/a0b54e19-f881-4c26-ad6c-958622064475/state
21762024-09-24T04:40:28.571ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47288
21772024-09-24T04:40:28.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47288 req_id = fa23f05a-ecdd-4db7-9767-69723cea104b response_code = 200 uri = /vpc-routes
21782024-09-24T04:40:28.593ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38631
21792024-09-24T04:40:28.593ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38631 req_id = aa1ac08c-d32d-49a5-a597-2e05882d6358 response_code = 200 uri = /vmms/a0b54e19-f881-4c26-ad6c-958622064475/state
21802024-09-24T04:40:28.595ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47288 req_id = 9c566af4-155e-4083-8e5b-d69e3aa54534 response_code = 204 uri = /vpc-routes
21812024-09-24T04:40:28.731ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57981
21822024-09-24T04:40:28.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57981 req_id = 3ce5d9c5-870a-4dca-b75f-24b7a72d0d4b response_code = 200 uri = /v2p
21832024-09-24T04:40:28.768ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36214
21842024-09-24T04:40:28.768ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36214 req_id = ab1bab52-024b-4b99-8e9a-2be0df7730b7 response_code = 200 uri = /vpc-routes
21852024-09-24T04:40:28.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36214 req_id = 03329ea4-8d15-484f-9130-bfcd2280a09f response_code = 204 uri = /vpc-routes
21862024-09-24T04:40:28.803ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46191
21872024-09-24T04:40:28.803ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
21882024-09-24T04:40:28.844ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
21892024-09-24T04:40:28.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97885 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46191 req_id = a3e22db1-60b6-4ee3-836a-624ef72981cc response_code = 204 uri = /switch-ports
21902024-09-24T04:40:28.901ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50754
21912024-09-24T04:40:28.901ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61533
21922024-09-24T04:40:28.901ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
21932024-09-24T04:40:28.905ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61533 req_id = 8dcb6f3f-173a-490a-a98b-77b4b98380f7 response_code = 200 uri = /v2p
21942024-09-24T04:40:28.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46191 req_id = 06636dd1-eb2a-4865-96ef-c9267c6809ae response_code = 200 uri = /network-bootstore-config
21952024-09-24T04:40:28.936ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
21962024-09-24T04:40:28.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92366 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50754 req_id = 24c32cd9-0273-4646-b456-9ff65398ff49 response_code = 204 uri = /switch-ports
21972024-09-24T04:40:28.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50754 req_id = 06ca5794-0fb6-4d61-82bd-5fee7c2ebf39 response_code = 200 uri = /network-bootstore-config
21982024-09-24T04:40:32.284ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35550
21992024-09-24T04:40:32.285ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1289 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35550 req_id = 04005469-af07-4f82-a73c-d2d1dd0656c6 response_code = 200 uri = /vmms/a0b54e19-f881-4c26-ad6c-958622064475/state
22002024-09-24T04:40:32.285ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-09-24T04:40:32.285645868Z } file = sled-agent/src/instance.rs:594 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22012024-09-24T04:40:32.285ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-24T04:40:32.285645868Z } propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22022024-09-24T04:40:32.285ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-24T04:40:32.285645868Z }, migration_in: None, migration_out: None }
22032024-09-24T04:40:32.327ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-09-24T04:40:32.327068318Z } file = sled-agent/src/instance.rs:594 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22042024-09-24T04:40:32.327ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-09-24T04:40:32.327068318Z } propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22052024-09-24T04:40:32.327ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:630 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22062024-09-24T04:40:32.327ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22072024-09-24T04:40:32.327ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 10509972363125552036 n_samples = 6 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
22082024-09-24T04:40:32.327ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22092024-09-24T04:40:32.327ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/1933fe24-deab-4311-a300-a09ef207e36a/debug/bundle/zone", "/pool/int/1cd7b3c4-f332-469d-a6b7-9f7501e7549c/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/036ed427-877d-4d45-ad05-e0fad8a2b1c2/crypt/debug", "/pool/ext/0f5ed56b-52e5-409d-bf51-72e9cba37d51/crypt/debug", "/pool/ext/d2bf0aa9-ae14-4c2a-ba8a-bd542e34f0ef/crypt/debug", "/pool/ext/7a7d72d1-c18a-4053-8666-72377ed40c63/crypt/debug", "/pool/ext/cf2b5393-7938-4187-81ff-eefd54b8dca3/crypt/debug", "/pool/ext/aa8dd6da-a2be-4c64-ab22-604023652cfd/crypt/debug", "/pool/ext/a09f232d-c31e-4961-bb00-74c2dac76055/crypt/debug", "/pool/ext/f46073e5-0cf7-4937-b820-b3dac0544ced/crypt/debug", "/pool/ext/5bcd6732-e8a3-402d-925d-d2e58792effe/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475
22102024-09-24T04:40:32.327ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 13149803485038243308 n_samples = 6 sled_id = 223823d0-6342-4be2-8483-3bbd7919ea67
22112024-09-24T04:40:32.961ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475", bundle_id: f56654e5-f161-4e41-808b-a0a9935d788e }, time_created: 2024-09-24T04:40:32.327944497Z, version: 0, cause: TerminatedInstance }
22122024-09-24T04:40:32.961ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_a0b54e19-f881-4c26-ad6c-958622064475 file = sled-agent/src/instance.rs:843 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22132024-09-24T04:40:34.724ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22142024-09-24T04:40:34.744ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(558280), kind: System }
22152024-09-24T04:40:34.744ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(558280), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
22162024-09-24T04:40:34.744ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-24T04:40:34.744525263Z }, migration_in: None, migration_out: None }
22172024-09-24T04:40:34.795ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-24T04:40:34.744525263Z }, migration_in: None, migration_out: None }
22182024-09-24T04:40:34.799ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:780 instance_id = 78d2b282-95f0-4b45-a7c2-e9d0874345dc propolis_id = a0b54e19-f881-4c26-ad6c-958622064475
22192024-09-24T04:40:35.174ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44353
22202024-09-24T04:40:35.174ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44353 req_id = 9715cff2-2112-4a1c-9ab0-466c44c2aa62 response_code = 200 uri = /vpc-routes
22212024-09-24T04:40:35.185ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 827 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44353 req_id = 8bee90b6-0841-4615-b830-4a77d0fe6f91 response_code = 204 uri = /vpc-routes
22222024-09-24T04:40:35.341ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52349
22232024-09-24T04:40:35.341ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 32 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52349 req_id = d0430dba-b942-4791-bc17-7a257ddda397 response_code = 200 uri = /baseboard
22242024-09-24T04:40:35.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47679
22252024-09-24T04:40:35.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47679 req_id = 6976ad04-00ac-4076-b2d5-8969c650041a response_code = 200 uri = /v2p
22262024-09-24T04:40:35.497ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43282
22272024-09-24T04:40:35.498ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43282 req_id = 96cfca0b-15f7-4c6f-9e94-2c026b308635 response_code = 200 uri = /v2p
22282024-09-24T04:40:36.192ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40782
22292024-09-24T04:40:36.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40782 req_id = fd953ce9-9e71-4026-8a41-4bf5b585313b response_code = 200 uri = /vpc-routes
22302024-09-24T04:40:36.201ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40782 req_id = f97a0d71-199e-4180-b621-a9fd461ce202 response_code = 204 uri = /vpc-routes