Max level shown:
1 [ Sep 11 22:48:33 Enabled. ]
2 [ Sep 11 22:48:33 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Sep 11 22:48:33 Method "start" exited with status 0. ]
4 [ Sep 11 22:48:33 Rereading configuration. ]
5 [ Sep 11 22:48:33 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-09-11T22:48:33.440ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-09-11T22:48:33.440ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-09-11T22:48:33.492ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-09-11T22:48:33.492ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-09-11T22:48:33.607ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-09-11T22:48:33.617ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-09-11T22:48:33.625ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-09-11T22:48:34.228ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-09-11T22:48:34.369ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-09-11T22:48:34.369ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-09-11T22:48:34.369ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-09-11T22:48:34.369ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-09-11T22:48:34.369ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-09-11T22:48:34.369ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-09-11T22:48:34.372ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-09-11T22:48:34.372ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-09-11T22:48:34.372ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-09-11T22:48:34.372ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-09-11T22:48:34.372ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-09-11T22:48:34.372ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-09-11T22:48:34.372ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-09-11T22:48:34.372ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-09-11T22:48:34.372ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-09-11T22:48:34.383ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-09-11T22:48:34.383ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool)
322024-09-11T22:48:34.482ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal }
332024-09-11T22:48:34.887ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-09-11T22:48:34.887ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-09-11T22:48:34.887ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-09-11T22:48:34.887ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-09-11T22:48:34.887ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-09-11T22:48:34.887ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-09-11T22:48:34.887ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-09-11T22:48:34.896ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-09-11T22:48:34.896ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool)
422024-09-11T22:48:34.983ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal }
432024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-09-11T22:48:35.391ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-09-11T22:48:35.391ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
532024-09-11T22:48:35.391ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
542024-09-11T22:48:35.391ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-09-11T22:48:35.391ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
582024-09-11T22:48:35.391ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
602024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-09-11T22:48:35.391ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-09-11T22:48:35.391ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-09-11T22:48:35.391ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-09-11T22:48:35.391ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-09-11T22:48:35.391ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
752024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
762024-09-11T22:48:35.391ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
772024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
782024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-09-11T22:48:35.391ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
822024-09-11T22:48:35.391ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-09-11T22:48:35.391ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-09-11T22:48:35.391ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-09-11T22:48:35.391ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872024-09-11T22:48:35.392ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
882024-09-11T22:48:35.392ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
892024-09-11T22:48:35.392ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
902024-09-11T22:48:35.392ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-09-11T22:48:35.392ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-09-11T22:48:35.392ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
932024-09-11T22:48:35.392ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
942024-09-11T22:48:35.392ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
952024-09-11T22:48:35.392ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
962024-09-11T22:48:35.392ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-09-11T22:48:35.392ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-09-11T22:48:35.392ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-09-11T22:48:35.392ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-09-11T22:48:35.392ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-09-11T22:48:35.392ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-09-11T22:48:35.392ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1032024-09-11T22:48:35.392ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-09-11T22:48:35.392ZINFOSledAgent (bootstore): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-09-11T22:48:35.392ZINFOSledAgent (bootstore): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-09-11T22:48:35.392ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-09-11T22:48:35.392ZINFOSledAgent (bootstore): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-09-11T22:48:35.392ZINFOSledAgent (bootstore): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-09-11T22:48:35.392ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-09-11T22:48:35.392ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-09-11T22:48:35.392ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-09-11T22:48:35.403ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:730
1132024-09-11T22:48:35.403ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1081, tv_nsec: 444103353 } time_to_next_cleanup = 599.999932289s
1142024-09-11T22:48:35.403ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3842
1152024-09-11T22:48:35.403ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-09-11T22:48:35.403ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-09-11T22:48:35.405ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-09-11T22:48:35.406ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-09-11T22:48:35.406ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-09-11T22:48:35.406ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-09-11T22:48:35.406ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1222024-09-11T22:48:35.406ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-09-11T22:48:35.406ZINFOSledAgent (RSS): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-09-11T22:48:35.406ZINFOSledAgent (RSS): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-09-11T22:48:35.406ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-09-11T22:48:35.406ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-09-11T22:48:35.406ZINFOSledAgent (RSS): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-09-11T22:48:35.406ZINFOSledAgent (RSS): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-09-11T22:48:35.406ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-09-11T22:48:35.406ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1312024-09-11T22:48:35.406ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1322024-09-11T22:48:35.406ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-09-11T22:48:35.406ZINFOSledAgent (RSS): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-09-11T22:48:35.407ZINFOSledAgent (RSS): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-09-11T22:48:35.407ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: ad186af8-8c6c-4691-b5b3-63d42bf4113c, sleds: { [fdb0:18c0:4d0d:a02a::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 35545f26-e1d9-4ad2-8ed3-679cba94411b, rack_id: ad186af8-8c6c-4691-b5b3-63d42bf4113c, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1362024-09-11T22:48:35.407ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-09-11T22:48:35.407ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-09-11T22:48:35.407ZINFOSledAgent (bootstore): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-09-11T22:48:35.407ZINFOSledAgent (bootstore): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-09-11T22:48:35.407ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-09-11T22:48:35.407ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:221 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 35545f26-e1d9-4ad2-8ed3-679cba94411b, rack_id: ad186af8-8c6c-4691-b5b3-63d42bf4113c, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-09-11T22:48:35.408ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:64240
1432024-09-11T22:48:35.408ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 35545f26-e1d9-4ad2-8ed3-679cba94411b, rack_id: ad186af8-8c6c-4691-b5b3-63d42bf4113c, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-09-11T22:48:35.408ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1462024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1472024-09-11T22:48:35.408ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1482024-09-11T22:48:35.408ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1492024-09-11T22:48:35.408ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1502024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1512024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1522024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1532024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1542024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1552024-09-11T22:48:35.408ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1562024-09-11T22:48:35.408ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1572024-09-11T22:48:35.410ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4139
1582024-09-11T22:48:35.410ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4521
1592024-09-11T22:48:35.440ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1602024-09-11T22:48:35.462ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-09-11T22:48:35.479ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-09-11T22:48:35.489ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:390 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1632024-09-11T22:48:35.489ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1642024-09-11T22:48:35.489ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:402 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1652024-09-11T22:48:35.489ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-09-11T22:48:35.492ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 66.592057ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ea064555-e7e2-456b-b74f-c6652d16c532", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ea064555-e7e2-456b-b74f-c6652d16c532" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-09-11T22:48:35.502ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_69499c41-13ca-4d82-92e5-3c29ce6f733d/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-09-11T22:48:35.526ZINFOSledAgent: successfully created encrypted zvol: name="oxi_69499c41-13ca-4d82-92e5-3c29ce6f733d/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-09-11T22:48:35.526ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_69499c41-13ca-4d82-92e5-3c29ce6f733d/swap" file = sled-agent/src/swap_device.rs:96
1702024-09-11T22:48:35.531ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:417 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1712024-09-11T22:48:35.531ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:130
1722024-09-11T22:48:35.531ZINFOSledAgent (BackingFs): Ensuring dataset oxi_69499c41-13ca-4d82-92e5-3c29ce6f733d/backing/fmd file = sled-agent/src/backing_fs.rs:140
1732024-09-11T22:48:35.559ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-09-11T22:48:35.560ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 63.147043ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1050bd35-b961-4cab-8f44-729c0e8e2a9f", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1050bd35-b961-4cab-8f44-729c0e8e2a9f" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-09-11T22:48:35.617ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:175
1762024-09-11T22:48:35.624ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-09-11T22:48:35.624ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 79.016221ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "608dbd9c-d1ea-4574-9667-42629054fc97", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "608dbd9c-d1ea-4574-9667-42629054fc97" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-09-11T22:48:35.705ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-09-11T22:48:35.705ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 97.3573ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b6544582-ddf7-495c-81ab-d91486f665ca", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b6544582-ddf7-495c-81ab-d91486f665ca" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-09-11T22:48:35.803ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1812024-09-11T22:48:35.804ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 188.038225ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9459648c-798b-4ca2-9e65-3421f706777d", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9459648c-798b-4ca2-9e65-3421f706777d" } file = clients/ddm-admin-client/src/lib.rs:177
1822024-09-11T22:48:35.836ZINFOSledAgent (BackingFs): Mounting oxi_69499c41-13ca-4d82-92e5-3c29ce6f733d/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:183
1832024-09-11T22:48:35.849ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:191
1842024-09-11T22:48:35.849ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:191
1852024-09-11T22:48:35.849ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:191
1862024-09-11T22:48:35.849ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:197
1872024-09-11T22:48:35.993ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-09-11T22:48:35.994ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 143.500341ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "20493036-04b7-4681-8628-88ec97ee0e78", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "20493036-04b7-4681-8628-88ec97ee0e78" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-09-11T22:48:36.138ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1902024-09-11T22:48:36.139ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 319.569671ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6153760c-236b-413a-80c2-53abacabc9f0", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6153760c-236b-413a-80c2-53abacabc9f0" } file = clients/ddm-admin-client/src/lib.rs:177
1912024-09-11T22:48:36.459ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1922024-09-11T22:48:36.460ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 258.647864ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a7f2b9fc-a894-46d9-98b1-7c03f5b9333c", "content-length": "124", "date": "Wed, 11 Sep 2024 22:48:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a7f2b9fc-a894-46d9-98b1-7c03f5b9333c" } file = clients/ddm-admin-client/src/lib.rs:177
1932024-09-11T22:48:36.582ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1942024-09-11T22:48:36.587ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:62558 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1952024-09-11T22:48:36.587ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1962024-09-11T22:48:36.587ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:62558 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1972024-09-11T22:48:36.589ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 145.765774ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1982024-09-11T22:48:36.589ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
1992024-09-11T22:48:36.589ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2002024-09-11T22:48:36.589ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:914 underlay_address = fd00:1122:3344:101::1
2012024-09-11T22:48:36.589ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
2022024-09-11T22:48:36.719ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2032024-09-11T22:48:36.737ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 527.970449ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2042024-09-11T22:48:37.267ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 722.821256ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2052024-09-11T22:48:37.990ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.683010821s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2062024-09-11T22:48:38.970ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2615
2072024-09-11T22:48:38.970ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2833
2082024-09-11T22:48:38.970ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2485
2092024-09-11T22:48:38.970ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2532
2102024-09-11T22:48:38.970ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2914
2112024-09-11T22:48:38.970ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2956
2122024-09-11T22:48:38.972ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2758
2132024-09-11T22:48:38.973ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2526
2142024-09-11T22:48:38.973ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='c185d971-4a81-4165-98a6-a3bf6a23e4dc'/> <propval type="astring" name="rack_id" value='ad186af8-8c6c-4691-b5b3-63d42bf4113c'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fef5:b750'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='35545f26-e1d9-4ad2-8ed3-679cba94411b'/> <propval type="astring" name="rack_id" value='ad186af8-8c6c-4691-b5b3-63d42bf4113c'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='35545f26-e1d9-4ad2-8ed3-679cba94411b'/> <propval type="astring" name="rack_uuid" value='ad186af8-8c6c-4691-b5b3-63d42bf4113c'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='35545f26-e1d9-4ad2-8ed3-679cba94411b'/> <propval type="astring" name="rack_uuid" value='ad186af8-8c6c-4691-b5b3-63d42bf4113c'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2152024-09-11T22:48:38.973ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2162024-09-11T22:48:39.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 58.090318ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2172024-09-11T22:48:39.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 85.90841ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2182024-09-11T22:48:39.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 261.714369ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-09-11T22:48:40.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 481.452693ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-09-11T22:48:40.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.069488693s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-09-11T22:48:40.688ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.183196897s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2222024-09-11T22:48:41.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.135542459s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-09-11T22:48:42.873ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 7.234209136s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2242024-09-11T22:48:43.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.003678484s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-09-11T22:48:44.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 628.326421ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-09-11T22:48:45.275ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2272024-09-11T22:48:45.275ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2282024-09-11T22:48:45.275ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2292024-09-11T22:48:45.275ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2302024-09-11T22:48:45.275ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2312024-09-11T22:48:45.276ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2322024-09-11T22:48:45.276ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:64240
2332024-09-11T22:48:45.276ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2342024-09-11T22:48:45.276ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3842
2352024-09-11T22:48:45.276ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:112 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2362024-09-11T22:48:45.276ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:236 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2372024-09-11T22:48:45.489ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2382024-09-11T22:48:45.489ZINFOSledAgent (RSS): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2392024-09-11T22:48:45.489ZINFOSledAgent (RSS): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2402024-09-11T22:48:45.489ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-09-11T22:48:45.489ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2422024-09-11T22:48:45.489ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2432024-09-11T22:48:45.494ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4163 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2442024-09-11T22:48:45.494ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4179
2452024-09-11T22:48:45.494ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2462024-09-11T22:48:45.746ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50532
2472024-09-11T22:48:45.746ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2482024-09-11T22:48:45.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 324 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50532 req_id = c0bcbfff-15e8-422a-b0ef-412f415de2d9 response_code = 200 uri = /inventory
2492024-09-11T22:48:45.778ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44795
2502024-09-11T22:48:45.781ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2552 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44795 req_id = 85b6c930-b01a-4f08-b73f-dcdf512c1a56 response_code = 200 uri = /sled-role
2512024-09-11T22:48:45.782ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2522024-09-11T22:48:45.782ZINFOSledAgent (RSS): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2532024-09-11T22:48:45.783ZINFOSledAgent (RSS): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2542024-09-11T22:48:45.783ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:1001
2552024-09-11T22:48:45.813ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: f0b18823-9e70-418a-80e8-42be4025d6e1, pool_id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 3f81eeec-e8f5-4666-89d9-81ccee68ee60, pool_id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 2f9eef08-45ea-4a23-9df4-f244beebebe7, pool_id: f7719a03-4894-4441-833a-ae918464a197 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 4b63a6a5-2ce6-489f-9d1d-3dbb680562f8, pool_id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 0c6c1236-5f5b-4f8a-a1f4-81dc7a6e9ce1, pool_id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: ec10ef59-4933-486c-a7fb-232557bc7d53, pool_id: e8902165-fb11-46b2-a335-df247ca55626 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 172fd876-74de-4ef5-935c-58c6384e236d, pool_id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: a99ce2de-1458-4e7e-96a0-83486486c338, pool_id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 8cb7b2b4-3301-4bf6-a1a5-323f4062c8fe, pool_id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2562024-09-11T22:48:45.813ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55135
2572024-09-11T22:48:45.813ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:858 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2582024-09-11T22:48:45.814ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: f0b18823-9e70-418a-80e8-42be4025d6e1, pool_id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 3f81eeec-e8f5-4666-89d9-81ccee68ee60, pool_id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 2f9eef08-45ea-4a23-9df4-f244beebebe7, pool_id: f7719a03-4894-4441-833a-ae918464a197 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 4b63a6a5-2ce6-489f-9d1d-3dbb680562f8, pool_id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 0c6c1236-5f5b-4f8a-a1f4-81dc7a6e9ce1, pool_id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: ec10ef59-4933-486c-a7fb-232557bc7d53, pool_id: e8902165-fb11-46b2-a335-df247ca55626 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 172fd876-74de-4ef5-935c-58c6384e236d, pool_id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: a99ce2de-1458-4e7e-96a0-83486486c338, pool_id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 8cb7b2b4-3301-4bf6-a1a5-323f4062c8fe, pool_id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2592024-09-11T22:48:45.814ZINFOSledAgent (StorageManager): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2602024-09-11T22:48:45.814ZINFOSledAgent (StorageManager): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2612024-09-11T22:48:45.814ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2622024-09-11T22:48:45.814ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2632024-09-11T22:48:45.814ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2642024-09-11T22:48:45.814ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2652024-09-11T22:48:45.814ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2662024-09-11T22:48:45.983ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2672024-09-11T22:48:45.983ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool)
2682024-09-11T22:48:46.437ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/35545f26-e1d9-4ad2-8ed3-679cba94411b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2692024-09-11T22:48:46.882ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2702024-09-11T22:48:46.912ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }
2712024-09-11T22:48:46.921ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:229
2722024-09-11T22:48:46.921ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:231
2732024-09-11T22:48:46.921ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:235
2742024-09-11T22:48:46.921ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:237
2752024-09-11T22:48:46.921ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2762024-09-11T22:48:46.921ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2772024-09-11T22:48:46.921ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
2782024-09-11T22:48:46.955ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2792024-09-11T22:48:46.963ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7102c444-682c-4072-8e1e-1a1f1e88f906/crypt/zone file = sled-storage/src/dataset.rs:292
2802024-09-11T22:48:47.139ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7102c444-682c-4072-8e1e-1a1f1e88f906 file = sled-storage/src/dataset.rs:378
2812024-09-11T22:48:47.147ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2822024-09-11T22:48:47.147ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2832024-09-11T22:48:47.147ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2842024-09-11T22:48:47.147ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2852024-09-11T22:48:47.147ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2862024-09-11T22:48:47.158ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2872024-09-11T22:48:47.158ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool)
2882024-09-11T22:48:47.250ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }
2892024-09-11T22:48:47.259ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:229
2902024-09-11T22:48:47.259ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:231
2912024-09-11T22:48:47.259ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:235
2922024-09-11T22:48:47.259ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:237
2932024-09-11T22:48:47.259ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2942024-09-11T22:48:47.259ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2952024-09-11T22:48:47.259ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
2962024-09-11T22:48:47.296ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2972024-09-11T22:48:47.304ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e/crypt/zone file = sled-storage/src/dataset.rs:292
2982024-09-11T22:48:47.440ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/35545f26-e1d9-4ad2-8ed3-679cba94411b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
2992024-09-11T22:48:47.475ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e file = sled-storage/src/dataset.rs:378
3002024-09-11T22:48:47.483ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3012024-09-11T22:48:47.483ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3022024-09-11T22:48:47.483ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3032024-09-11T22:48:47.483ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3042024-09-11T22:48:47.483ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3052024-09-11T22:48:47.493ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3062024-09-11T22:48:47.493ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = f7719a03-4894-4441-833a-ae918464a197 (zpool)
3072024-09-11T22:48:47.589ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }
3082024-09-11T22:48:47.597ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:229
3092024-09-11T22:48:47.597ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:231
3102024-09-11T22:48:47.597ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:235
3112024-09-11T22:48:47.597ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:237
3122024-09-11T22:48:47.597ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3132024-09-11T22:48:47.597ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3142024-09-11T22:48:47.597ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3152024-09-11T22:48:47.630ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3162024-09-11T22:48:47.638ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f7719a03-4894-4441-833a-ae918464a197/crypt/zone file = sled-storage/src/dataset.rs:292
3172024-09-11T22:48:47.801ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f7719a03-4894-4441-833a-ae918464a197 file = sled-storage/src/dataset.rs:378
3182024-09-11T22:48:47.808ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3192024-09-11T22:48:47.808ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3202024-09-11T22:48:47.808ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3212024-09-11T22:48:47.809ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3222024-09-11T22:48:47.809ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3232024-09-11T22:48:47.818ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3242024-09-11T22:48:47.818ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool)
3252024-09-11T22:48:47.912ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }
3262024-09-11T22:48:47.920ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:229
3272024-09-11T22:48:47.920ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:231
3282024-09-11T22:48:47.920ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:235
3292024-09-11T22:48:47.920ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:237
3302024-09-11T22:48:47.920ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3312024-09-11T22:48:47.920ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3322024-09-11T22:48:47.920ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3332024-09-11T22:48:47.954ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3342024-09-11T22:48:47.961ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_d700c1e0-2586-4b27-b46c-83bfe687db08/crypt/zone file = sled-storage/src/dataset.rs:292
3352024-09-11T22:48:48.123ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_d700c1e0-2586-4b27-b46c-83bfe687db08 file = sled-storage/src/dataset.rs:378
3362024-09-11T22:48:48.130ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3372024-09-11T22:48:48.130ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3382024-09-11T22:48:48.130ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3392024-09-11T22:48:48.130ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3402024-09-11T22:48:48.130ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3412024-09-11T22:48:48.139ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3422024-09-11T22:48:48.139ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 8f9c145d-f66d-4533-97ae-72e50686935e (zpool)
3432024-09-11T22:48:48.234ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }
3442024-09-11T22:48:48.241ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:229
3452024-09-11T22:48:48.241ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:231
3462024-09-11T22:48:48.241ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:235
3472024-09-11T22:48:48.241ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:237
3482024-09-11T22:48:48.241ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3492024-09-11T22:48:48.242ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3502024-09-11T22:48:48.242ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3512024-09-11T22:48:48.275ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3522024-09-11T22:48:48.283ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8f9c145d-f66d-4533-97ae-72e50686935e/crypt/zone file = sled-storage/src/dataset.rs:292
3532024-09-11T22:48:48.443ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/35545f26-e1d9-4ad2-8ed3-679cba94411b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
3542024-09-11T22:48:48.459ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8f9c145d-f66d-4533-97ae-72e50686935e file = sled-storage/src/dataset.rs:378
3552024-09-11T22:48:48.467ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3562024-09-11T22:48:48.467ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3572024-09-11T22:48:48.467ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3582024-09-11T22:48:48.467ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3592024-09-11T22:48:48.467ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3602024-09-11T22:48:48.477ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3612024-09-11T22:48:48.477ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = e8902165-fb11-46b2-a335-df247ca55626 (zpool)
3622024-09-11T22:48:48.572ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: e8902165-fb11-46b2-a335-df247ca55626 (zpool), kind: External }
3632024-09-11T22:48:48.580ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:229
3642024-09-11T22:48:48.580ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:231
3652024-09-11T22:48:48.580ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:235
3662024-09-11T22:48:48.580ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:237
3672024-09-11T22:48:48.580ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3682024-09-11T22:48:48.580ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3692024-09-11T22:48:48.580ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3702024-09-11T22:48:48.616ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3712024-09-11T22:48:48.623ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e8902165-fb11-46b2-a335-df247ca55626/crypt/zone file = sled-storage/src/dataset.rs:292
3722024-09-11T22:48:48.785ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e8902165-fb11-46b2-a335-df247ca55626 file = sled-storage/src/dataset.rs:378
3732024-09-11T22:48:48.792ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3742024-09-11T22:48:48.792ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3752024-09-11T22:48:48.792ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3762024-09-11T22:48:48.792ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3772024-09-11T22:48:48.792ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3782024-09-11T22:48:48.802ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3792024-09-11T22:48:48.802ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool)
3802024-09-11T22:48:48.895ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool), kind: External }
3812024-09-11T22:48:48.905ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:229
3822024-09-11T22:48:48.906ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:231
3832024-09-11T22:48:48.906ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:235
3842024-09-11T22:48:48.906ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:237
3852024-09-11T22:48:48.906ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3862024-09-11T22:48:48.906ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3872024-09-11T22:48:48.906ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3882024-09-11T22:48:48.942ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3892024-09-11T22:48:48.952ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7408a7f2-57c1-401a-a865-fdbf6a508b61/crypt/zone file = sled-storage/src/dataset.rs:292
3902024-09-11T22:48:49.133ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7408a7f2-57c1-401a-a865-fdbf6a508b61 file = sled-storage/src/dataset.rs:378
3912024-09-11T22:48:49.141ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3922024-09-11T22:48:49.141ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3932024-09-11T22:48:49.141ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3942024-09-11T22:48:49.141ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3952024-09-11T22:48:49.141ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3962024-09-11T22:48:49.151ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3972024-09-11T22:48:49.151ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = cd45e164-586a-42ad-8f77-733daff8a747 (zpool)
3982024-09-11T22:48:49.247ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool), kind: External }
3992024-09-11T22:48:49.255ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:229
4002024-09-11T22:48:49.255ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:231
4012024-09-11T22:48:49.255ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:235
4022024-09-11T22:48:49.255ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:237
4032024-09-11T22:48:49.255ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4042024-09-11T22:48:49.255ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4052024-09-11T22:48:49.255ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
4062024-09-11T22:48:49.290ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4072024-09-11T22:48:49.298ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cd45e164-586a-42ad-8f77-733daff8a747/crypt/zone file = sled-storage/src/dataset.rs:292
4082024-09-11T22:48:49.445ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/35545f26-e1d9-4ad2-8ed3-679cba94411b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4092024-09-11T22:48:49.483ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cd45e164-586a-42ad-8f77-733daff8a747 file = sled-storage/src/dataset.rs:378
4102024-09-11T22:48:49.492ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4112024-09-11T22:48:49.492ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4122024-09-11T22:48:49.492ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4132024-09-11T22:48:49.492ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4142024-09-11T22:48:49.492ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4152024-09-11T22:48:49.501ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4162024-09-11T22:48:49.501ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool)
4172024-09-11T22:48:49.594ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External }
4182024-09-11T22:48:49.601ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:229
4192024-09-11T22:48:49.602ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:231
4202024-09-11T22:48:49.602ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:235
4212024-09-11T22:48:49.602ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:237
4222024-09-11T22:48:49.602ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4232024-09-11T22:48:49.602ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4242024-09-11T22:48:49.602ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
4252024-09-11T22:48:49.639ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4262024-09-11T22:48:49.647ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2fc72b15-af8d-491c-9d35-a084b85733a3/crypt/zone file = sled-storage/src/dataset.rs:292
4272024-09-11T22:48:49.821ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2fc72b15-af8d-491c-9d35-a084b85733a3 file = sled-storage/src/dataset.rs:378
4282024-09-11T22:48:49.829ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4292024-09-11T22:48:49.829ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4302024-09-11T22:48:49.829ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4312024-09-11T22:48:49.830ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 69499c41-13ca-4d82-92e5-3c29ce6f733d (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 43105ead-3497-4a46-b5a2-e1fdd8df799c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: e8902165-fb11-46b2-a335-df247ca55626 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4322024-09-11T22:48:49.830ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4332024-09-11T22:48:49.830ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4342024-09-11T22:48:49.830ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4352024-09-11T22:48:49.830ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:862 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4362024-09-11T22:48:49.830ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4372024-09-11T22:48:49.830ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:888 generation = Generation(1) sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4382024-09-11T22:48:49.830ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:893 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4392024-09-11T22:48:49.830ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:898 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4402024-09-11T22:48:49.830ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4412024-09-11T22:48:49.830ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:904 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4422024-09-11T22:48:49.830ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:803 gen = Generation(1)
4432024-09-11T22:48:49.830ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:909 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4442024-09-11T22:48:49.830ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4016846 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55135 req_id = 4566c9e6-b957-4b74-96e0-158324cdb449 response_code = 200 uri = /omicron-physical-disks
4452024-09-11T22:48:49.862ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4462024-09-11T22:48:49.862ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41592
4472024-09-11T22:48:49.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4482024-09-11T22:48:49.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4492024-09-11T22:48:49.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4502024-09-11T22:48:49.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
4512024-09-11T22:48:49.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4522024-09-11T22:48:49.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4532024-09-11T22:48:50.010ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4542024-09-11T22:48:50.010ZINFOSledAgent (ServiceManager): No ledger in /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:188
4552024-09-11T22:48:50.010ZINFOSledAgent (ServiceManager): No ledger in /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:188
4562024-09-11T22:48:50.010ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4572024-09-11T22:48:50.051ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4582024-09-11T22:48:50.051ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4592024-09-11T22:48:50.051ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4602024-09-11T22:48:50.105ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044 file = illumos-utils/src/zone.rs:316
4612024-09-11T22:48:50.109ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928 file = illumos-utils/src/zone.rs:316
4622024-09-11T22:48:50.112ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 22.83831553s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4632024-09-11T22:48:50.113ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7 file = illumos-utils/src/zone.rs:316
4642024-09-11T22:48:50.125ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044 file = illumos-utils/src/zone.rs:354
4652024-09-11T22:48:50.130ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928 file = illumos-utils/src/zone.rs:354
4662024-09-11T22:48:50.134ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7 file = illumos-utils/src/zone.rs:354
4672024-09-11T22:48:50.447ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/35545f26-e1d9-4ad2-8ed3-679cba94411b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
4682024-09-11T22:48:51.582ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4692024-09-11T22:48:51.582ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4702024-09-11T22:48:51.582ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4712024-09-11T22:48:52.066ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4722024-09-11T22:48:52.549ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4732024-09-11T22:48:52.549ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4742024-09-11T22:48:52.549ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4752024-09-11T22:48:52.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 28.762811ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4762024-09-11T22:48:52.557ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4772024-09-11T22:48:52.561ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4782024-09-11T22:48:52.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 122.227182ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4792024-09-11T22:48:52.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 38.886877ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4802024-09-11T22:48:52.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 54.461608ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4812024-09-11T22:48:52.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 108.389134ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4822024-09-11T22:48:52.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 92.665706ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4832024-09-11T22:48:52.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 63.23982ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4842024-09-11T22:48:52.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 266.668285ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4852024-09-11T22:48:53.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 295.36748ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4862024-09-11T22:48:53.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 104.081306ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4872024-09-11T22:48:53.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 381.025993ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4882024-09-11T22:48:53.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 409.906922ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4892024-09-11T22:48:53.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 343.067803ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4902024-09-11T22:48:53.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 401.993886ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4912024-09-11T22:48:53.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 1.0178883s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4922024-09-11T22:48:53.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 579.861933ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4932024-09-11T22:48:54.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 644.56817ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4942024-09-11T22:48:54.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 936.22302ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4952024-09-11T22:48:54.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 933.225207ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
4962024-09-11T22:48:54.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 1.350743616s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
4972024-09-11T22:48:55.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 780.998862ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
4982024-09-11T22:48:55.518ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4187
4992024-09-11T22:48:55.518ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4197 gateway = fd00:1122:3344:101::1
5002024-09-11T22:48:55.523ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4276
5012024-09-11T22:48:55.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 1.223533822s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
5022024-09-11T22:48:55.645ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4324
5032024-09-11T22:48:55.645ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4351
5042024-09-11T22:48:55.685ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4364
5052024-09-11T22:48:55.685ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4228
5062024-09-11T22:48:55.738ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4270
5072024-09-11T22:48:55.738ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4331 rack_subnet = fd00:1122:3344::
5082024-09-11T22:48:55.757ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4342
5092024-09-11T22:48:55.757ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4384
5102024-09-11T22:48:55.845ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4416
5112024-09-11T22:48:55.845ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4422
5122024-09-11T22:48:55.952ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4463
5132024-09-11T22:48:55.953ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5142024-09-11T22:48:55.986ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/35545f26-e1d9-4ad2-8ed3-679cba94411b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:571 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
5152024-09-11T22:48:55.986ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 53.165937ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5162024-09-11T22:48:56.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 550.425532ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
5172024-09-11T22:48:56.044ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 96.055094ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5182024-09-11T22:48:56.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044") failed: Property not found. retry in 1.162399214s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
5192024-09-11T22:48:56.142ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5202024-09-11T22:48:56.173ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:42717
5212024-09-11T22:48:56.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:42717 req_id = 92836644-ce51-4b6a-84c9-5c8f43c77eed response_code = 200 uri = /sled-identifiers
5222024-09-11T22:48:56.176ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5232024-09-11T22:48:56.176ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5242024-09-11T22:48:56.176ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5252024-09-11T22:48:56.176ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5262024-09-11T22:48:56.213ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
5272024-09-11T22:48:56.262ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5282024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5292024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5302024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5312024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5322024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5332024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5342024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5352024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5362024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5372024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5382024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5392024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5402024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5412024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5422024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5432024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5442024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5452024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5462024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5472024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5482024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5492024-09-11T22:48:56.262ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5502024-09-11T22:48:56.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928") failed: Property not found. retry in 1.357159935s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
5512024-09-11T22:48:56.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7") failed: Property not found. retry in 869.302264ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
5522024-09-11T22:48:57.983ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_682c62dd-94d7-4139-a150-aabd86949044
5532024-09-11T22:48:57.983ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_049af1e3-bb38-4a4b-a180-bf5fe0ec9928
5542024-09-11T22:48:57.983ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_internal_dns_2f744a48-842b-4847-9f91-b1611298b0c7
5552024-09-11T22:48:57.983ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5562024-09-11T22:48:57.983ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5572024-09-11T22:48:57.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8120955 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41592 req_id = 086c7feb-b1f0-4929-90a3-1045290b36ce response_code = 204 uri = /omicron-zones
5582024-09-11T22:48:57.986ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5592024-09-11T22:48:58.036ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5602024-09-11T22:48:58.066ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 93.554062ms) dns_config_addr = [fd00:1122:3344:2::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:2::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:595
5612024-09-11T22:48:58.178ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5622024-09-11T22:48:58.225ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5632024-09-11T22:48:58.225ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5642024-09-11T22:48:58.256ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5652024-09-11T22:48:58.257ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5662024-09-11T22:48:58.257ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5672024-09-11T22:48:58.287ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5682024-09-11T22:48:58.287ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52351
5692024-09-11T22:48:58.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5702024-09-11T22:48:58.288ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5712024-09-11T22:48:58.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5722024-09-11T22:48:58.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
5732024-09-11T22:48:58.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5742024-09-11T22:48:58.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5752024-09-11T22:48:58.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5762024-09-11T22:48:58.346ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
5772024-09-11T22:48:58.346ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
5782024-09-11T22:48:58.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5792024-09-11T22:48:58.357ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5802024-09-11T22:48:58.390ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5812024-09-11T22:48:58.392ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5822024-09-11T22:48:58.392ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5832024-09-11T22:48:58.423ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 175, 181]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5842024-09-11T22:48:58.423ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = boundary_ntp
5852024-09-11T22:48:58.424ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
5862024-09-11T22:48:58.424ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
5872024-09-11T22:48:58.424ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
5882024-09-11T22:48:58.426ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5892024-09-11T22:48:58.450ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17 file = illumos-utils/src/zone.rs:316
5902024-09-11T22:48:58.470ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17 file = illumos-utils/src/zone.rs:354
5912024-09-11T22:48:59.101ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5922024-09-11T22:48:59.101ZINFOSledAgent (ServiceManager): Booting oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
5932024-09-11T22:48:59.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 53.195062ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
5942024-09-11T22:48:59.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 96.544597ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
5952024-09-11T22:48:59.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 280.301153ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
5962024-09-11T22:48:59.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 413.774021ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
5972024-09-11T22:49:00.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 865.868978ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
5982024-09-11T22:49:01.009ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 967.089713ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
5992024-09-11T22:49:01.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 1.025177965s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
6002024-09-11T22:49:03.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17") failed: Property not found. retry in 1.16317585s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
6012024-09-11T22:49:04.204ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17
6022024-09-11T22:49:04.204ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6032024-09-11T22:49:04.204ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6042024-09-11T22:49:04.204ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5917220 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52351 req_id = d102d2e0-faee-4dc2-9f72-347a542de4ef response_code = 204 uri = /omicron-zones
6052024-09-11T22:49:04.207ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6062024-09-11T22:49:04.236ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6072024-09-11T22:49:04.237ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54391
6082024-09-11T22:49:04.246ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6092024-09-11T22:49:04.246ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 9316 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54391 req_id = 28678b53-6c3e-4ac0-99df-31893494749f response_code = 500 uri = /timesync
6102024-09-11T22:49:04.248ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6112024-09-11T22:49:04.430ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6122024-09-11T22:49:04.430ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53524
6132024-09-11T22:49:08.441ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6142024-09-11T22:49:08.441ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4011167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53524 req_id = 99eaf606-c32a-4067-8948-3d0c31848996 response_code = 500 uri = /timesync
6152024-09-11T22:49:08.443ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6162024-09-11T22:49:08.577ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6172024-09-11T22:49:08.577ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57384
6182024-09-11T22:49:08.586ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6192024-09-11T22:49:08.586ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8505 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57384 req_id = 17740339-4365-4740-aaec-a88fc2e724f4 response_code = 500 uri = /timesync
6202024-09-11T22:49:08.587ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6212024-09-11T22:49:08.710ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6222024-09-11T22:49:08.710ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49706
6232024-09-11T22:49:12.421ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4f7368ab-da2e-43f0-ad3c-9661a8944f17': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3828
6242024-09-11T22:49:12.421ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4008118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49706 req_id = 1f7d1478-7557-4dda-bd67-ba0468e7715e response_code = 500 uri = /timesync
6252024-09-11T22:49:12.425ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6262024-09-11T22:49:12.601ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6272024-09-11T22:49:12.601ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60838
6282024-09-11T22:49:12.612ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 10421 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60838 req_id = 77c7b4ac-1f92-487a-8878-aeb27d5c5984 response_code = 200 uri = /timesync
6292024-09-11T22:49:12.614ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1726094948.3542922, correction: 0.039063893 } file = sled-agent/src/rack_setup/service.rs:660
6302024-09-11T22:49:12.614ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6312024-09-11T22:49:12.839ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6322024-09-11T22:49:12.839ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57915
6332024-09-11T22:49:12.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8688 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57915 req_id = 49b40fae-2c00-4043-9a41-422b9e372a32 response_code = 200 uri = /timesync
6342024-09-11T22:49:12.850ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1726094948.3542922, correction: 0.038425833 } file = sled-agent/src/rack_setup/service.rs:660
6352024-09-11T22:49:12.850ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6362024-09-11T22:49:13.111ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6372024-09-11T22:49:13.111ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48741
6382024-09-11T22:49:13.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7863 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48741 req_id = a5af2064-d4ed-4401-ba61-5950e45e1251 response_code = 200 uri = /timesync
6392024-09-11T22:49:13.121ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1726094948.3542922, correction: 0.03769327 } file = sled-agent/src/rack_setup/service.rs:660
6402024-09-11T22:49:13.121ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6412024-09-11T22:49:13.518ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6422024-09-11T22:49:13.518ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47449
6432024-09-11T22:49:13.538ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 20172 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47449 req_id = 36f26290-b045-4348-ba07-c6d8cca2d811 response_code = 200 uri = /timesync
6442024-09-11T22:49:13.541ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 1757546723, ip_addr: 104.194.8.227, stratum: 3, ref_time: 1726094953.212428, correction: 0.036952984 } file = sled-agent/src/rack_setup/service.rs:660
6452024-09-11T22:49:13.541ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1299
6462024-09-11T22:49:13.573ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: 13966b4e-ce93-437d-a312-454f1cac159b, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6472024-09-11T22:49:13.573ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55122
6482024-09-11T22:49:13.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6492024-09-11T22:49:13.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6502024-09-11T22:49:13.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6512024-09-11T22:49:13.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
6522024-09-11T22:49:13.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6532024-09-11T22:49:13.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6542024-09-11T22:49:13.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6552024-09-11T22:49:13.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6562024-09-11T22:49:13.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6572024-09-11T22:49:13.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }
6582024-09-11T22:49:13.739ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6592024-09-11T22:49:13.739ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6602024-09-11T22:49:13.799ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6612024-09-11T22:49:13.799ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6622024-09-11T22:49:13.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6632024-09-11T22:49:13.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }
6642024-09-11T22:49:13.900ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6652024-09-11T22:49:13.900ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
6662024-09-11T22:49:13.900ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
6672024-09-11T22:49:13.901ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6682024-09-11T22:49:13.933ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
6692024-09-11T22:49:13.933ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
6702024-09-11T22:49:13.933ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
6712024-09-11T22:49:13.933ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
6722024-09-11T22:49:13.933ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
6732024-09-11T22:49:13.933ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556", TimeNotSynchronized), ("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b", TimeNotSynchronized), ("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369", TimeNotSynchronized), ("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0", TimeNotSynchronized), ("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 358427 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55122 req_id = 92fdb1a2-2082-4666-8e2e-9874c1ea5128 response_code = 500 uri = /omicron-zones
6742024-09-11T22:49:13.933ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "92fdb1a2-2082-4666-8e2e-9874c1ea5128", "content-length": "124", "date": "Wed, 11 Sep 2024 22:49:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "92fdb1a2-2082-4666-8e2e-9874c1ea5128" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 81.101653ms sled_address = [fd00:1122:3344:101::1]:12345
6752024-09-11T22:49:14.015ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: 13966b4e-ce93-437d-a312-454f1cac159b, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6762024-09-11T22:49:14.017ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6772024-09-11T22:49:14.017ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6782024-09-11T22:49:14.035ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6792024-09-11T22:49:14.035ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
6802024-09-11T22:49:14.053ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6812024-09-11T22:49:14.053ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6822024-09-11T22:49:14.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6832024-09-11T22:49:14.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6842024-09-11T22:49:14.090ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6852024-09-11T22:49:14.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }
6862024-09-11T22:49:14.109ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6872024-09-11T22:49:14.109ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6882024-09-11T22:49:14.127ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6892024-09-11T22:49:14.127ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6902024-09-11T22:49:14.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6912024-09-11T22:49:14.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }
6922024-09-11T22:49:14.163ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6932024-09-11T22:49:14.163ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
6942024-09-11T22:49:14.164ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
6952024-09-11T22:49:14.164ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6962024-09-11T22:49:14.195ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
6972024-09-11T22:49:14.196ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
6982024-09-11T22:49:14.196ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
6992024-09-11T22:49:14.196ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
7002024-09-11T22:49:14.196ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
7012024-09-11T22:49:14.196ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369", TimeNotSynchronized), ("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556", TimeNotSynchronized), ("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15", TimeNotSynchronized), ("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b", TimeNotSynchronized), ("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 179457 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55122 req_id = 5e0de369-9a29-4dce-8df2-fa7b30ebd696 response_code = 500 uri = /omicron-zones
7022024-09-11T22:49:14.196ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5e0de369-9a29-4dce-8df2-fa7b30ebd696", "content-length": "124", "date": "Wed, 11 Sep 2024 22:49:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5e0de369-9a29-4dce-8df2-fa7b30ebd696" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 84.745185ms sled_address = [fd00:1122:3344:101::1]:12345
7032024-09-11T22:49:14.281ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: 13966b4e-ce93-437d-a312-454f1cac159b, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7042024-09-11T22:49:14.281ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7052024-09-11T22:49:14.281ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7062024-09-11T22:49:14.300ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7072024-09-11T22:49:14.300ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
7082024-09-11T22:49:14.317ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7092024-09-11T22:49:14.317ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7102024-09-11T22:49:14.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7112024-09-11T22:49:14.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7122024-09-11T22:49:14.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7132024-09-11T22:49:14.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }
7142024-09-11T22:49:14.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7152024-09-11T22:49:14.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7162024-09-11T22:49:14.388ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7172024-09-11T22:49:14.388ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7182024-09-11T22:49:14.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7192024-09-11T22:49:14.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }
7202024-09-11T22:49:14.426ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7212024-09-11T22:49:14.426ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
7222024-09-11T22:49:14.426ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
7232024-09-11T22:49:14.426ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7242024-09-11T22:49:14.458ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
7252024-09-11T22:49:14.458ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
7262024-09-11T22:49:14.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
7272024-09-11T22:49:14.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
7282024-09-11T22:49:14.459ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
7292024-09-11T22:49:14.459ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369", TimeNotSynchronized), ("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15", TimeNotSynchronized), ("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0", TimeNotSynchronized), ("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b", TimeNotSynchronized), ("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 176877 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55122 req_id = 10649087-623c-4eaf-bf6c-3e6338374b9c response_code = 500 uri = /omicron-zones
7302024-09-11T22:49:14.459ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "10649087-623c-4eaf-bf6c-3e6338374b9c", "content-length": "124", "date": "Wed, 11 Sep 2024 22:49:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "10649087-623c-4eaf-bf6c-3e6338374b9c" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 76.939111ms sled_address = [fd00:1122:3344:101::1]:12345
7312024-09-11T22:49:14.538ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: 13966b4e-ce93-437d-a312-454f1cac159b, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7322024-09-11T22:49:14.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7332024-09-11T22:49:14.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7342024-09-11T22:49:14.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7352024-09-11T22:49:14.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
7362024-09-11T22:49:14.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7372024-09-11T22:49:14.575ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7382024-09-11T22:49:14.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7392024-09-11T22:49:14.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7402024-09-11T22:49:14.612ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7412024-09-11T22:49:14.612ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }
7422024-09-11T22:49:14.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7432024-09-11T22:49:14.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7442024-09-11T22:49:14.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7452024-09-11T22:49:14.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7462024-09-11T22:49:14.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7472024-09-11T22:49:14.668ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }
7482024-09-11T22:49:14.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7492024-09-11T22:49:14.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
7502024-09-11T22:49:14.685ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
7512024-09-11T22:49:14.685ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7522024-09-11T22:49:14.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
7532024-09-11T22:49:14.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
7542024-09-11T22:49:14.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
7552024-09-11T22:49:14.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
7562024-09-11T22:49:14.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
7572024-09-11T22:49:14.715ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0", TimeNotSynchronized), ("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b", TimeNotSynchronized), ("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15", TimeNotSynchronized), ("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556", TimeNotSynchronized), ("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 176682 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55122 req_id = 1b91aebd-d297-4374-a01c-811b5b0033f8 response_code = 500 uri = /omicron-zones
7582024-09-11T22:49:14.715ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1b91aebd-d297-4374-a01c-811b5b0033f8", "content-length": "124", "date": "Wed, 11 Sep 2024 22:49:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1b91aebd-d297-4374-a01c-811b5b0033f8" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 186.310182ms sled_address = [fd00:1122:3344:101::1]:12345
7592024-09-11T22:49:14.903ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: 13966b4e-ce93-437d-a312-454f1cac159b, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7602024-09-11T22:49:14.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7612024-09-11T22:49:14.903ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7622024-09-11T22:49:14.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7632024-09-11T22:49:14.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
7642024-09-11T22:49:14.939ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7652024-09-11T22:49:14.939ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7662024-09-11T22:49:14.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7672024-09-11T22:49:14.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7682024-09-11T22:49:14.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7692024-09-11T22:49:14.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }
7702024-09-11T22:49:14.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7712024-09-11T22:49:14.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7722024-09-11T22:49:15.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7732024-09-11T22:49:15.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7742024-09-11T22:49:15.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7752024-09-11T22:49:15.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }
7762024-09-11T22:49:15.047ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7772024-09-11T22:49:15.047ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
7782024-09-11T22:49:15.047ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
7792024-09-11T22:49:15.047ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7802024-09-11T22:49:15.080ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
7812024-09-11T22:49:15.080ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
7822024-09-11T22:49:15.080ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
7832024-09-11T22:49:15.080ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
7842024-09-11T22:49:15.080ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3230 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
7852024-09-11T22:49:15.080ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0", TimeNotSynchronized), ("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556", TimeNotSynchronized), ("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369", TimeNotSynchronized), ("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b", TimeNotSynchronized), ("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 176752 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55122 req_id = c60e8e6b-0b69-4c15-abdd-777ea013bba2 response_code = 500 uri = /omicron-zones
7862024-09-11T22:49:15.080ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c60e8e6b-0b69-4c15-abdd-777ea013bba2", "content-length": "124", "date": "Wed, 11 Sep 2024 22:49:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c60e8e6b-0b69-4c15-abdd-777ea013bba2" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 211.611724ms sled_address = [fd00:1122:3344:101::1]:12345
7872024-09-11T22:49:15.294ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: 13966b4e-ce93-437d-a312-454f1cac159b, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7882024-09-11T22:49:15.295ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7892024-09-11T22:49:15.295ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7902024-09-11T22:49:15.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7912024-09-11T22:49:15.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
7922024-09-11T22:49:15.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7932024-09-11T22:49:15.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7942024-09-11T22:49:15.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7952024-09-11T22:49:15.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7962024-09-11T22:49:15.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7972024-09-11T22:49:15.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }
7982024-09-11T22:49:15.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7992024-09-11T22:49:15.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8002024-09-11T22:49:15.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8012024-09-11T22:49:15.403ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8022024-09-11T22:49:15.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8032024-09-11T22:49:15.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }
8042024-09-11T22:49:15.439ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8052024-09-11T22:49:15.439ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
8062024-09-11T22:49:15.439ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
8072024-09-11T22:49:15.439ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8082024-09-11T22:49:15.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8092024-09-11T22:49:15.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8102024-09-11T22:49:15.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8112024-09-11T22:49:15.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8122024-09-11T22:49:15.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8132024-09-11T22:49:15.611ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0 file = illumos-utils/src/zone.rs:316
8142024-09-11T22:49:15.615ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556 file = illumos-utils/src/zone.rs:316
8152024-09-11T22:49:15.619ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15 file = illumos-utils/src/zone.rs:316
8162024-09-11T22:49:15.623ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b file = illumos-utils/src/zone.rs:316
8172024-09-11T22:49:15.628ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369 file = illumos-utils/src/zone.rs:316
8182024-09-11T22:49:15.632ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0 file = illumos-utils/src/zone.rs:354
8192024-09-11T22:49:15.637ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556 file = illumos-utils/src/zone.rs:354
8202024-09-11T22:49:15.641ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15 file = illumos-utils/src/zone.rs:354
8212024-09-11T22:49:15.646ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b file = illumos-utils/src/zone.rs:354
8222024-09-11T22:49:15.650ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369 file = illumos-utils/src/zone.rs:354
8232024-09-11T22:49:18.652ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='13966b4e-ce93-437d-a312-454f1cac159b'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8242024-09-11T22:49:18.652ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8252024-09-11T22:49:18.657ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='81a2d138-4b9c-46b9-b4d2-d87d97946556'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8262024-09-11T22:49:18.657ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8272024-09-11T22:49:18.708ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8282024-09-11T22:49:18.708ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8292024-09-11T22:49:18.717ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='bc976e87-bbc5-436b-9be1-3ca0f70c0ad0'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8302024-09-11T22:49:18.717ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8312024-09-11T22:49:18.758ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8322024-09-11T22:49:18.759ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8332024-09-11T22:49:18.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 68.794808ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8342024-09-11T22:49:18.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 44.827898ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8352024-09-11T22:49:18.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 67.609252ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8362024-09-11T22:49:18.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 43.464774ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8372024-09-11T22:49:19.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 61.595012ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8382024-09-11T22:49:19.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 91.712384ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8392024-09-11T22:49:19.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 74.537409ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8402024-09-11T22:49:19.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 40.665982ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8412024-09-11T22:49:20.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 144.562221ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8422024-09-11T22:49:20.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 110.960663ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8432024-09-11T22:49:20.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 293.472794ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8442024-09-11T22:49:20.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 193.098274ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8452024-09-11T22:49:20.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 126.665087ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8462024-09-11T22:49:20.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 487.459541ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8472024-09-11T22:49:20.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 269.817101ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8482024-09-11T22:49:21.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 392.21185ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8492024-09-11T22:49:21.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 464.284526ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8502024-09-11T22:49:21.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 121.183935ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8512024-09-11T22:49:21.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 535.573473ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8522024-09-11T22:49:21.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.144412501s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8532024-09-11T22:49:21.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 402.62859ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8542024-09-11T22:49:21.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 1.017293373s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8552024-09-11T22:49:21.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 1.017368935s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8562024-09-11T22:49:21.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 831.681096ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8572024-09-11T22:49:21.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.018537513s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8582024-09-11T22:49:22.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 727.411096ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8592024-09-11T22:49:22.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 1.13587881s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8602024-09-11T22:49:22.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 1.200062385s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8612024-09-11T22:49:22.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 780.367395ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8622024-09-11T22:49:22.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 858.920769ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8632024-09-11T22:49:23.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.302215739s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8642024-09-11T22:49:23.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 999.555711ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8652024-09-11T22:49:23.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 1.390463405s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8662024-09-11T22:49:23.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 910.688119ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8672024-09-11T22:49:23.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 779.812155ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8682024-09-11T22:49:24.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 712.539248ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8692024-09-11T22:49:24.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 1.281325802s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8702024-09-11T22:49:24.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 1.384047473s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8712024-09-11T22:49:24.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 748.730237ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8722024-09-11T22:49:25.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 572.201604ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8732024-09-11T22:49:25.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 966.841511ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8742024-09-11T22:49:25.459ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46154
8752024-09-11T22:49:25.460ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 798 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46154 req_id = ded2ef2e-ecd1-4309-9b1e-3dface0e8c8f response_code = 200 uri = /baseboard
8762024-09-11T22:49:25.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 591.556498ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8772024-09-11T22:49:25.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 555.092092ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8782024-09-11T22:49:25.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 1.012828682s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8792024-09-11T22:49:26.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.437755125s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8802024-09-11T22:49:26.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 1.381883925s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8812024-09-11T22:49:26.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 1.197976469s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8822024-09-11T22:49:26.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.403653417s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8832024-09-11T22:49:26.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556") failed: Property not found. retry in 1.178135127s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
8842024-09-11T22:49:27.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0") failed: Property not found. retry in 500.079324ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
8852024-09-11T22:49:27.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15") failed: Property not found. retry in 1.302524361s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
8862024-09-11T22:49:27.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.212164527s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8872024-09-11T22:49:27.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.063138797s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8882024-09-11T22:49:27.733ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 211.857928ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
8892024-09-11T22:49:28.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.399415311s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8902024-09-11T22:49:28.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.051376321s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8912024-09-11T22:49:29.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.108693074s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8922024-09-11T22:49:30.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.390214205s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8932024-09-11T22:49:30.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 768.442309ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8942024-09-11T22:49:31.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.337594184s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8952024-09-11T22:49:31.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 949.351653ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8962024-09-11T22:49:32.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 748.81791ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8972024-09-11T22:49:32.972ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 736.118909ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
8982024-09-11T22:49:33.502ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.307710405s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
8992024-09-11T22:49:33.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.446322196s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
9002024-09-11T22:49:34.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 512.980338ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
9012024-09-11T22:49:35.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 711.380715ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
9022024-09-11T22:49:35.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 1.111538645s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
9032024-09-11T22:49:35.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 950.954456ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
9042024-09-11T22:49:36.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 991.447593ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
9052024-09-11T22:49:36.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 895.782158ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
9062024-09-11T22:49:37.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b") failed: Property not found. retry in 522.333537ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
9072024-09-11T22:49:37.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369") failed: Property not found. retry in 1.42222797s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
9082024-09-11T22:49:39.208ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_bc976e87-bbc5-436b-9be1-3ca0f70c0ad0
9092024-09-11T22:49:39.209ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_81a2d138-4b9c-46b9-b4d2-d87d97946556
9102024-09-11T22:49:39.209ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15
9112024-09-11T22:49:39.209ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_13966b4e-ce93-437d-a312-454f1cac159b
9122024-09-11T22:49:39.209ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_cockroachdb_e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369
9132024-09-11T22:49:39.209ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9142024-09-11T22:49:39.209ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9152024-09-11T22:49:39.210ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 23883719 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55122 req_id = c0d42029-daa1-457c-89c2-a1a81be68cfd response_code = 204 uri = /omicron-zones
9162024-09-11T22:49:39.247ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55201
9172024-09-11T22:49:39.280ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:3662
9182024-09-11T22:49:42.923ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3680
9192024-09-11T22:49:54.094ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3703
9202024-09-11T22:49:54.094ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 14846666 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:55201 req_id = 271c77d2-13d1-4737-a15b-d8f0508f8924 response_code = 204 uri = /cockroachdb
9212024-09-11T22:49:54.098ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 464.45117ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
9222024-09-11T22:49:54.129ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 682c62dd-94d7-4139-a150-aabd86949044, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2f744a48-842b-4847-9f91-b1611298b0c7, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 57986a59-1242-4d0a-9c97-abaa0f11b3ab, kind: Service { id: 4f7368ab-da2e-43f0-ad3c-9661a8944f17 }, name: Name("ntp-4f7368ab-da2e-43f0-ad3c-9661a8944f17"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 181])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: 13966b4e-ce93-437d-a312-454f1cac159b, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }, OmicronZoneConfig { id: a9318a9c-1482-41a4-ba3e-a515b62e612b, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 60c011c6-d26c-45f9-887c-70fa53b449b0, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c39300d-43ce-4e36-8a27-0fe8091f4f2f, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6366d8dc-d4d9-43fe-8e81-251bf7f28abd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: e8902165-fb11-46b2-a335-df247ca55626 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: bbde0be4-b1a6-44b2-9c97-d73f27be6718, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: adffb264-4c4d-41de-9bc2-166e2d0d5e82, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External } } } }, OmicronZoneConfig { id: d36fee4e-6ff8-4647-a0bf-5944c7102e9f, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External } } } }, OmicronZoneConfig { id: b8d92285-3041-49be-9542-cbf9c8a2e3e0, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External } } } }, OmicronZoneConfig { id: 64ce5226-a6ee-4705-8a1c-d6a6d9056277, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External } } } }, OmicronZoneConfig { id: abe48a0b-4221-441c-9356-29c0d26548ef, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External } } } }, OmicronZoneConfig { id: 75ac48f4-97b5-446c-bf76-d6d4c5e42f80, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: e8902165-fb11-46b2-a335-df247ca55626 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e8902165-fb11-46b2-a335-df247ca55626 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9f252058-a33d-4f45-86c5-bc171de68d56, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3c490ca9-6a8f-485c-b3a2-e1e8571725e9, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5fbf461d-9a9b-46ff-acc7-244b7fd6b244, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9232024-09-11T22:49:54.129ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52951
9242024-09-11T22:49:54.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9252024-09-11T22:49:54.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 682c62dd-94d7-4139-a150-aabd86949044, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9262024-09-11T22:49:54.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9272024-09-11T22:49:54.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 049af1e3-bb38-4a4b-a180-bf5fe0ec9928, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: InternalDns }, responder: ... }
9282024-09-11T22:49:54.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9292024-09-11T22:49:54.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2f744a48-842b-4847-9f91-b1611298b0c7, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9302024-09-11T22:49:54.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9312024-09-11T22:49:54.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a5d778b3-a4fe-458a-ac1b-a7cf1f5dfb15, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9322024-09-11T22:49:54.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9332024-09-11T22:49:54.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 81a2d138-4b9c-46b9-b4d2-d87d97946556, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Cockroach }, responder: ... }
9342024-09-11T22:49:54.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9352024-09-11T22:49:54.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 13966b4e-ce93-437d-a312-454f1cac159b, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9362024-09-11T22:49:54.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9372024-09-11T22:49:54.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: bc976e87-bbc5-436b-9be1-3ca0f70c0ad0, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9382024-09-11T22:49:54.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9392024-09-11T22:49:54.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e50bceaf-8bb9-4d7a-aeaf-8a2db5adf369, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Cockroach }, responder: ... }
9402024-09-11T22:49:54.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9318a9c-1482-41a4-ba3e-a515b62e612b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9412024-09-11T22:49:54.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a9318a9c-1482-41a4-ba3e-a515b62e612b, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9422024-09-11T22:49:54.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9432024-09-11T22:49:54.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9442024-09-11T22:49:54.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c39300d-43ce-4e36-8a27-0fe8091f4f2f, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
9452024-09-11T22:49:54.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 8c39300d-43ce-4e36-8a27-0fe8091f4f2f, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9462024-09-11T22:49:54.582ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9472024-09-11T22:49:54.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d, dataset_name: DatasetName { pool_name: ZpoolName { id: 7102c444-682c-4072-8e1e-1a1f1e88f906 (zpool), kind: External }, kind: Crucible }, responder: ... }
9482024-09-11T22:49:54.678ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d36fee4e-6ff8-4647-a0bf-5944c7102e9f, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9492024-09-11T22:49:54.678ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: d36fee4e-6ff8-4647-a0bf-5944c7102e9f, dataset_name: DatasetName { pool_name: ZpoolName { id: ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e (zpool), kind: External }, kind: Crucible }, responder: ... }
9502024-09-11T22:49:54.765ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8d92285-3041-49be-9542-cbf9c8a2e3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9512024-09-11T22:49:54.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: b8d92285-3041-49be-9542-cbf9c8a2e3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: f7719a03-4894-4441-833a-ae918464a197 (zpool), kind: External }, kind: Crucible }, responder: ... }
9522024-09-11T22:49:54.851ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64ce5226-a6ee-4705-8a1c-d6a6d9056277, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9532024-09-11T22:49:54.851ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 64ce5226-a6ee-4705-8a1c-d6a6d9056277, dataset_name: DatasetName { pool_name: ZpoolName { id: d700c1e0-2586-4b27-b46c-83bfe687db08 (zpool), kind: External }, kind: Crucible }, responder: ... }
9542024-09-11T22:49:54.941ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: abe48a0b-4221-441c-9356-29c0d26548ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9552024-09-11T22:49:54.941ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: abe48a0b-4221-441c-9356-29c0d26548ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f9c145d-f66d-4533-97ae-72e50686935e (zpool), kind: External }, kind: Crucible }, responder: ... }
9562024-09-11T22:49:55.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 75ac48f4-97b5-446c-bf76-d6d4c5e42f80, dataset_name: DatasetName { pool_name: ZpoolName { id: e8902165-fb11-46b2-a335-df247ca55626 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9572024-09-11T22:49:55.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 75ac48f4-97b5-446c-bf76-d6d4c5e42f80, dataset_name: DatasetName { pool_name: ZpoolName { id: e8902165-fb11-46b2-a335-df247ca55626 (zpool), kind: External }, kind: Crucible }, responder: ... }
9582024-09-11T22:49:55.122ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9f252058-a33d-4f45-86c5-bc171de68d56, dataset_name: DatasetName { pool_name: ZpoolName { id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9592024-09-11T22:49:55.122ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9f252058-a33d-4f45-86c5-bc171de68d56, dataset_name: DatasetName { pool_name: ZpoolName { id: 7408a7f2-57c1-401a-a865-fdbf6a508b61 (zpool), kind: External }, kind: Crucible }, responder: ... }
9602024-09-11T22:49:55.214ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c490ca9-6a8f-485c-b3a2-e1e8571725e9, dataset_name: DatasetName { pool_name: ZpoolName { id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9612024-09-11T22:49:55.214ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3c490ca9-6a8f-485c-b3a2-e1e8571725e9, dataset_name: DatasetName { pool_name: ZpoolName { id: cd45e164-586a-42ad-8f77-733daff8a747 (zpool), kind: External }, kind: Crucible }, responder: ... }
9622024-09-11T22:49:55.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5fbf461d-9a9b-46ff-acc7-244b7fd6b244, dataset_name: DatasetName { pool_name: ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9632024-09-11T22:49:55.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5fbf461d-9a9b-46ff-acc7-244b7fd6b244, dataset_name: DatasetName { pool_name: ZpoolName { id: 2fc72b15-af8d-491c-9d35-a084b85733a3 (zpool), kind: External }, kind: Crucible }, responder: ... }
9642024-09-11T22:49:55.368ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9652024-09-11T22:49:55.368ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
9662024-09-11T22:49:55.368ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
9672024-09-11T22:49:55.368ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9682024-09-11T22:49:55.500ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40446
9692024-09-11T22:49:55.500ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40446 req_id = e6e06658-98e1-423c-b021-dbc7b2308208 response_code = 200 uri = /baseboard
9702024-09-11T22:49:55.502ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9712024-09-11T22:49:55.502ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9722024-09-11T22:49:55.502ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9732024-09-11T22:49:55.502ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9742024-09-11T22:49:55.517ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9752024-09-11T22:49:55.517ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9762024-09-11T22:49:55.517ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9772024-09-11T22:49:55.529ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9782024-09-11T22:49:55.542ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9792024-09-11T22:49:55.542ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9802024-09-11T22:49:55.723ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9812024-09-11T22:49:55.723ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9822024-09-11T22:49:55.723ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9832024-09-11T22:49:55.723ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9842024-09-11T22:49:55.723ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9852024-09-11T22:49:55.724ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9862024-09-11T22:49:55.760ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9872024-09-11T22:49:55.760ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9882024-09-11T22:49:55.760ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9892024-09-11T22:49:55.760ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718 file = illumos-utils/src/zone.rs:316
9902024-09-11T22:49:55.766ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80 file = illumos-utils/src/zone.rs:316
9912024-09-11T22:49:55.772ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0 file = illumos-utils/src/zone.rs:316
9922024-09-11T22:49:55.778ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0 file = illumos-utils/src/zone.rs:316
9932024-09-11T22:49:55.783ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56 file = illumos-utils/src/zone.rs:316
9942024-09-11T22:49:55.789ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f file = illumos-utils/src/zone.rs:316
9952024-09-11T22:49:55.795ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd file = illumos-utils/src/zone.rs:316
9962024-09-11T22:49:55.938ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718 file = illumos-utils/src/zone.rs:354
9972024-09-11T22:49:55.944ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80 file = illumos-utils/src/zone.rs:354
9982024-09-11T22:49:55.948ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0 file = illumos-utils/src/zone.rs:354
9992024-09-11T22:49:55.952ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0 file = illumos-utils/src/zone.rs:354
10002024-09-11T22:49:55.957ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56 file = illumos-utils/src/zone.rs:354
10012024-09-11T22:49:55.962ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f file = illumos-utils/src/zone.rs:354
10022024-09-11T22:49:55.966ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd file = illumos-utils/src/zone.rs:354
10032024-09-11T22:49:55.971ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9 file = illumos-utils/src/zone.rs:316
10042024-09-11T22:49:55.976ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244 file = illumos-utils/src/zone.rs:316
10052024-09-11T22:49:55.982ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82 file = illumos-utils/src/zone.rs:316
10062024-09-11T22:49:55.987ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d file = illumos-utils/src/zone.rs:316
10072024-09-11T22:49:55.993ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f file = illumos-utils/src/zone.rs:316
10082024-09-11T22:49:55.999ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef file = illumos-utils/src/zone.rs:316
10092024-09-11T22:49:56.004ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277 file = illumos-utils/src/zone.rs:316
10102024-09-11T22:49:56.048ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10112024-09-11T22:49:56.049ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9 file = illumos-utils/src/zone.rs:354
10122024-09-11T22:49:56.055ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244 file = illumos-utils/src/zone.rs:354
10132024-09-11T22:49:56.062ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82 file = illumos-utils/src/zone.rs:354
10142024-09-11T22:49:56.068ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d file = illumos-utils/src/zone.rs:354
10152024-09-11T22:49:56.075ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f file = illumos-utils/src/zone.rs:354
10162024-09-11T22:49:56.080ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef file = illumos-utils/src/zone.rs:354
10172024-09-11T22:49:56.086ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277 file = illumos-utils/src/zone.rs:354
10182024-09-11T22:49:56.154ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10192024-09-11T22:49:56.254ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10202024-09-11T22:49:56.354ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10212024-09-11T22:49:56.357ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10222024-09-11T22:49:56.357ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10232024-09-11T22:49:56.417ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 143, 82]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10242024-09-11T22:49:56.417ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
10252024-09-11T22:49:56.481ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10262024-09-11T22:49:56.483ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10272024-09-11T22:49:56.483ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10282024-09-11T22:49:56.549ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 147, 229]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10292024-09-11T22:49:56.549ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
10302024-09-11T22:49:56.552ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10312024-09-11T22:49:56.552ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10322024-09-11T22:49:56.608ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 209, 23]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
10332024-09-11T22:49:56.608ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
10342024-09-11T22:49:56.610ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10352024-09-11T22:49:56.610ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10362024-09-11T22:49:56.664ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 197, 188]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
10372024-09-11T22:49:56.665ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
10382024-09-11T22:49:56.666ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10392024-09-11T22:49:56.666ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10402024-09-11T22:49:56.725ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 143, 246]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10412024-09-11T22:49:56.725ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
10422024-09-11T22:49:56.725ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
10432024-09-11T22:49:56.725ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
10442024-09-11T22:49:56.725ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
10452024-09-11T22:49:56.725ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
10462024-09-11T22:49:56.726ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
10472024-09-11T22:49:56.728ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10482024-09-11T22:49:56.785ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
10492024-09-11T22:49:56.790ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
10502024-09-11T22:49:56.790ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
10512024-09-11T22:49:56.790ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10522024-09-11T22:49:56.790ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
10532024-09-11T22:49:56.790ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
10542024-09-11T22:49:56.790ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
10552024-09-11T22:49:56.790ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
10562024-09-11T22:49:56.844ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
10572024-09-11T22:49:56.848ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
10582024-09-11T22:49:56.852ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
10592024-09-11T22:49:56.852ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10602024-09-11T22:49:56.852ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10612024-09-11T22:49:56.855ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10622024-09-11T22:49:56.855ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f file = illumos-utils/src/zone.rs:316
10632024-09-11T22:49:56.870ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62 file = illumos-utils/src/zone.rs:316
10642024-09-11T22:49:57.050ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f file = illumos-utils/src/zone.rs:354
10652024-09-11T22:49:57.058ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62 file = illumos-utils/src/zone.rs:354
10662024-09-11T22:49:57.064ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf file = illumos-utils/src/zone.rs:316
10672024-09-11T22:49:57.070ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b file = illumos-utils/src/zone.rs:316
10682024-09-11T22:49:57.076ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88 file = illumos-utils/src/zone.rs:316
10692024-09-11T22:49:57.095ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf file = illumos-utils/src/zone.rs:354
10702024-09-11T22:49:57.102ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b file = illumos-utils/src/zone.rs:354
10712024-09-11T22:49:57.108ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88 file = illumos-utils/src/zone.rs:354
10722024-09-11T22:49:59.133ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10732024-09-11T22:49:59.133ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
10742024-09-11T22:49:59.148ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10752024-09-11T22:49:59.148ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
10762024-09-11T22:49:59.318ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10772024-09-11T22:49:59.318ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
10782024-09-11T22:49:59.334ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='60c011c6-d26c-45f9-887c-70fa53b449b0'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10792024-09-11T22:49:59.335ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
10802024-09-11T22:49:59.343ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7408a7f2-57c1-401a-a865-fdbf6a508b61/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5019334d-d297-41b0-8c7d-4cdeac4144f1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10812024-09-11T22:49:59.343ZINFOSledAgent (ServiceManager): Booting oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
10822024-09-11T22:49:59.366ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e8902165-fb11-46b2-a335-df247ca55626/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='eecea0ba-5bf1-4cf8-b770-51ba44872f82'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10832024-09-11T22:49:59.366ZINFOSledAgent (ServiceManager): Booting oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
10842024-09-11T22:49:59.386ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f7719a03-4894-4441-833a-ae918464a197/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='321c0234-929a-4115-8b16-2a2bc5200de1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10852024-09-11T22:49:59.386ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
10862024-09-11T22:49:59.401ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7102c444-682c-4072-8e1e-1a1f1e88f906/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a3000e33-7466-4dde-8adc-e5c3cbfaf18a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10872024-09-11T22:49:59.401ZINFOSledAgent (ServiceManager): Booting oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
10882024-09-11T22:49:59.427ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_cd45e164-586a-42ad-8f77-733daff8a747/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='19e3974f-5747-445b-81c2-9800e09fe102'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10892024-09-11T22:49:59.427ZINFOSledAgent (ServiceManager): Booting oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
10902024-09-11T22:49:59.441ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_8f9c145d-f66d-4533-97ae-72e50686935e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='77a6b324-10a2-4a7b-a797-a8eb7f4c219a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10912024-09-11T22:49:59.441ZINFOSledAgent (ServiceManager): Booting oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
10922024-09-11T22:49:59.500ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_d700c1e0-2586-4b27-b46c-83bfe687db08/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='86e174c0-de88-4589-9512-0b1dc6d305d9'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10932024-09-11T22:49:59.500ZINFOSledAgent (ServiceManager): Booting oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
10942024-09-11T22:49:59.511ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_2fc72b15-af8d-491c-9d35-a084b85733a3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='c8142ae9-f2ca-4850-b0cd-db70d45c16de'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10952024-09-11T22:49:59.511ZINFOSledAgent (ServiceManager): Booting oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
10962024-09-11T22:49:59.685ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='f0891191-26fe-4de3-b272-6a38e6c262f1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10972024-09-11T22:49:59.686ZINFOSledAgent (ServiceManager): Booting oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
10982024-09-11T22:50:00.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 61.390839ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
10992024-09-11T22:50:00.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 28.460305ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11002024-09-11T22:50:00.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 74.22858ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11012024-09-11T22:50:00.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 27.854625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11022024-09-11T22:50:00.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 38.173465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11032024-09-11T22:50:00.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 34.160638ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11042024-09-11T22:50:00.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 63.766784ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11052024-09-11T22:50:00.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 57.360544ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11062024-09-11T22:50:00.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 66.625393ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11072024-09-11T22:50:00.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 30.558436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11082024-09-11T22:50:00.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 45.636059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11092024-09-11T22:50:00.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 67.661487ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
11102024-09-11T22:50:00.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 70.832335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
11112024-09-11T22:50:00.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 125.860685ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11122024-09-11T22:50:00.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 49.070305ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11132024-09-11T22:50:00.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 136.869021ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
11142024-09-11T22:50:00.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 124.465269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11152024-09-11T22:50:01.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 112.491103ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11162024-09-11T22:50:01.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 120.696236ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11172024-09-11T22:50:01.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 114.11058ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11182024-09-11T22:50:01.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 123.324731ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
11192024-09-11T22:50:01.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 137.558673ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
11202024-09-11T22:50:01.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 160.776707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11212024-09-11T22:50:01.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 83.891975ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11222024-09-11T22:50:01.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 118.254006ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11232024-09-11T22:50:01.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 142.748358ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11242024-09-11T22:50:01.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 110.687972ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11252024-09-11T22:50:01.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 66.335547ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11262024-09-11T22:50:01.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 194.842225ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
11272024-09-11T22:50:01.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 327.195533ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11282024-09-11T22:50:01.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 160.769349ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11292024-09-11T22:50:01.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 187.688178ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11302024-09-11T22:50:01.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 134.036401ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11312024-09-11T22:50:01.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 111.124626ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
11322024-09-11T22:50:01.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 297.513543ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
11332024-09-11T22:50:01.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 244.140164ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11342024-09-11T22:50:01.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 123.78805ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11352024-09-11T22:50:01.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 248.036707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11362024-09-11T22:50:01.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 154.008553ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11372024-09-11T22:50:01.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 131.576539ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11382024-09-11T22:50:01.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 502.567878ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
11392024-09-11T22:50:01.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.071409739s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11402024-09-11T22:50:01.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 327.684786ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11412024-09-11T22:50:01.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 493.056828ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11422024-09-11T22:50:01.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 245.694582ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11432024-09-11T22:50:01.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 404.521458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
11442024-09-11T22:50:01.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 500.019316ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
11452024-09-11T22:50:01.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 367.229977ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11462024-09-11T22:50:01.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 435.974258ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11472024-09-11T22:50:01.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 253.777158ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11482024-09-11T22:50:02.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 282.451713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11492024-09-11T22:50:02.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 391.176207ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11502024-09-11T22:50:02.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 497.214902ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11512024-09-11T22:50:02.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 654.172187ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11522024-09-11T22:50:02.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.194053874s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11532024-09-11T22:50:02.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.166378408s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11542024-09-11T22:50:03.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 623.83132ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11552024-09-11T22:50:03.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.411838911s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11562024-09-11T22:50:03.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 774.863391ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11572024-09-11T22:50:03.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 813.322294ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11582024-09-11T22:50:03.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 729.079745ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
11592024-09-11T22:50:03.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 564.527062ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
11602024-09-11T22:50:03.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 453.940024ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11612024-09-11T22:50:03.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 517.785213ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11622024-09-11T22:50:03.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.157540089s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11632024-09-11T22:50:03.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 427.554439ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11642024-09-11T22:50:03.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 940.617837ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
11652024-09-11T22:50:03.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.030158055s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11662024-09-11T22:50:03.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 774.596433ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11672024-09-11T22:50:03.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 886.513371ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11682024-09-11T22:50:03.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.156218985s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11692024-09-11T22:50:03.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 940.54521ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11702024-09-11T22:50:03.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.397076067s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
11712024-09-11T22:50:04.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.274074344s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11722024-09-11T22:50:04.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.49857794s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11732024-09-11T22:50:04.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.054428023s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11742024-09-11T22:50:04.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 924.968628ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
11752024-09-11T22:50:04.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 796.893096ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11762024-09-11T22:50:04.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 518.942992ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
11772024-09-11T22:50:04.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.394868404s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11782024-09-11T22:50:04.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 590.09703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11792024-09-11T22:50:04.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 660.876378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11802024-09-11T22:50:04.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 670.417285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11812024-09-11T22:50:04.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.339802786s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11822024-09-11T22:50:04.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 625.977359ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11832024-09-11T22:50:04.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.409563339s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
11842024-09-11T22:50:05.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.373049538s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
11852024-09-11T22:50:05.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.42029754s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
11862024-09-11T22:50:05.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 570.155054ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11872024-09-11T22:50:05.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.331609581s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
11882024-09-11T22:50:05.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.149919095s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
11892024-09-11T22:50:05.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.305832192s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
11902024-09-11T22:50:05.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 571.811394ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11912024-09-11T22:50:05.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.289833809s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
11922024-09-11T22:50:05.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 999.077668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
11932024-09-11T22:50:05.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 545.459333ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11942024-09-11T22:50:05.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.011826858s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
11952024-09-11T22:50:05.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.015650982s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
11962024-09-11T22:50:05.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.21877035s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
11972024-09-11T22:50:06.049ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.448143019s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
11982024-09-11T22:50:06.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 545.632732ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
11992024-09-11T22:50:06.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 782.430485ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
12002024-09-11T22:50:06.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.02209316s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
12012024-09-11T22:50:06.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 525.949749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
12022024-09-11T22:50:06.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.399166256s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
12032024-09-11T22:50:06.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 524.71425ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
12042024-09-11T22:50:06.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 926.495381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
12052024-09-11T22:50:06.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.345702919s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
12062024-09-11T22:50:06.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 721.149893ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
12072024-09-11T22:50:06.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 662.761275ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
12082024-09-11T22:50:06.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.460297334s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
12092024-09-11T22:50:06.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.099484592s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
12102024-09-11T22:50:07.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.496328229s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
12112024-09-11T22:50:07.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.272850969s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
12122024-09-11T22:50:07.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 852.756151ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
12132024-09-11T22:50:07.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 577.809885ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
12142024-09-11T22:50:07.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.294558361s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
12152024-09-11T22:50:07.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 871.309957ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
12162024-09-11T22:50:07.502ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 738.038313ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
12172024-09-11T22:50:07.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 503.725755ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
12182024-09-11T22:50:07.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 504.475831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
12192024-09-11T22:50:07.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.414452426s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
12202024-09-11T22:50:07.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 956.649059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
12212024-09-11T22:50:07.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 663.073207ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
12222024-09-11T22:50:08.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 646.066668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
12232024-09-11T22:50:08.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.305418543s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
12242024-09-11T22:50:08.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 935.750572ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
12252024-09-11T22:50:08.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.033902498s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
12262024-09-11T22:50:08.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.432802655s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
12272024-09-11T22:50:08.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.129924727s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
12282024-09-11T22:50:08.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.081452719s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
12292024-09-11T22:50:08.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 605.704028ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
12302024-09-11T22:50:08.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.093159046s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
12312024-09-11T22:50:08.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 768.016366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
12322024-09-11T22:50:08.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 710.789357ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
12332024-09-11T22:50:08.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 590.17813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
12342024-09-11T22:50:08.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.145040247s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
12352024-09-11T22:50:09.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 993.720424ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
12362024-09-11T22:50:09.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 952.162774ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
12372024-09-11T22:50:09.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.049255613s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
12382024-09-11T22:50:09.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.140305666s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
12392024-09-11T22:50:09.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.00381036s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
12402024-09-11T22:50:09.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 546.02466ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
12412024-09-11T22:50:09.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 907.881488ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
12422024-09-11T22:50:09.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.481078458s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
12432024-09-11T22:50:09.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 710.44198ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
12442024-09-11T22:50:09.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 652.976847ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
12452024-09-11T22:50:09.639ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 787.937018ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
12462024-09-11T22:50:09.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 555.302654ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
12472024-09-11T22:50:09.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 778.778852ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
12482024-09-11T22:50:09.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 577.10991ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
12492024-09-11T22:50:10.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.296127325s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
12502024-09-11T22:50:10.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 605.273981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
12512024-09-11T22:50:10.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 891.987514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
12522024-09-11T22:50:10.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.240258248s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
12532024-09-11T22:50:10.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 747.921316ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
12542024-09-11T22:50:10.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.38448795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
12552024-09-11T22:50:10.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.465941025s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
12562024-09-11T22:50:10.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.020768971s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
12572024-09-11T22:50:10.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.140904497s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
12582024-09-11T22:50:10.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.39434028s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
12592024-09-11T22:50:10.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 700.817477ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
12602024-09-11T22:50:10.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.175451601s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
12612024-09-11T22:50:10.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.480285798s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
12622024-09-11T22:50:10.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.400966505s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
12632024-09-11T22:50:10.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 901.703292ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
12642024-09-11T22:50:11.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 934.298613ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
12652024-09-11T22:50:11.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 684.214931ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
12662024-09-11T22:50:11.327ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12672024-09-11T22:50:11.328ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
12682024-09-11T22:50:11.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.060463376s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
12692024-09-11T22:50:11.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 685.069187ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
12702024-09-11T22:50:11.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 841.59782ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
12712024-09-11T22:50:11.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.319900995s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
12722024-09-11T22:50:11.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 25.624295ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
12732024-09-11T22:50:11.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.281898459s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
12742024-09-11T22:50:11.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 72.626002ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
12752024-09-11T22:50:11.671ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12762024-09-11T22:50:11.671ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
12772024-09-11T22:50:11.679ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12782024-09-11T22:50:11.679ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
12792024-09-11T22:50:11.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 870.243466ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
12802024-09-11T22:50:11.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 242.170675ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
12812024-09-11T22:50:11.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.33925678s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
12822024-09-11T22:50:11.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 633.496668ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
12832024-09-11T22:50:11.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 572.655665ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
12842024-09-11T22:50:11.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.073108145s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
12852024-09-11T22:50:11.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.399933278s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
12862024-09-11T22:50:11.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 60.376294ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
12872024-09-11T22:50:12.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 68.100656ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
12882024-09-11T22:50:12.003ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12892024-09-11T22:50:12.004ZINFOSledAgent (ServiceManager): Booting oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
12902024-09-11T22:50:12.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 582.464354ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
12912024-09-11T22:50:12.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 141.920495ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
12922024-09-11T22:50:12.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 647.033447ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
12932024-09-11T22:50:12.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 92.01932ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
12942024-09-11T22:50:12.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 852.245833ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
12952024-09-11T22:50:12.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 288.112126ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
12962024-09-11T22:50:12.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 113.421641ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
12972024-09-11T22:50:12.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 904.48196ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
12982024-09-11T22:50:12.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 62.911843ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
12992024-09-11T22:50:12.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.020043815s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
13002024-09-11T22:50:12.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 90.397845ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13012024-09-11T22:50:12.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 448.139913ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
13022024-09-11T22:50:12.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.392942874s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
13032024-09-11T22:50:12.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 255.546904ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13042024-09-11T22:50:12.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 607.306798ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
13052024-09-11T22:50:12.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.464037595s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
13062024-09-11T22:50:12.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 445.265422ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
13072024-09-11T22:50:12.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 588.295689ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
13082024-09-11T22:50:12.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 978.052652ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
13092024-09-11T22:50:12.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 447.168634ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13102024-09-11T22:50:12.725ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13112024-09-11T22:50:12.725ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13122024-09-11T22:50:12.725ZINFOSledAgent (ServiceManager): Booting oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13132024-09-11T22:50:12.731ZINFOSledAgent (ServiceManager): Booting oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13142024-09-11T22:50:12.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 735.312783ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
13152024-09-11T22:50:12.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.18179316s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
13162024-09-11T22:50:12.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 946.722184ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
13172024-09-11T22:50:12.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 587.962442ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
13182024-09-11T22:50:12.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 1.047524818s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
13192024-09-11T22:50:13.009ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 60.508771ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13202024-09-11T22:50:13.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 66.525727ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13212024-09-11T22:50:13.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.144547903s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
13222024-09-11T22:50:13.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.136219695s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
13232024-09-11T22:50:13.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 71.739967ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13242024-09-11T22:50:13.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 122.243361ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13252024-09-11T22:50:13.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 986.831997ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
13262024-09-11T22:50:13.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.256055602s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
13272024-09-11T22:50:13.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 249.713844ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13282024-09-11T22:50:13.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 688.442356ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13292024-09-11T22:50:13.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.386477016s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
13302024-09-11T22:50:13.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.269293902s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
13312024-09-11T22:50:13.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 167.018707ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13322024-09-11T22:50:13.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 812.30671ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
13332024-09-11T22:50:13.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 919.013612ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
13342024-09-11T22:50:13.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 244.426392ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13352024-09-11T22:50:13.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.121783554s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
13362024-09-11T22:50:13.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 231.008063ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13372024-09-11T22:50:13.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 801.459001ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
13382024-09-11T22:50:13.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 568.916225ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
13392024-09-11T22:50:13.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 488.901943ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13402024-09-11T22:50:13.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.013098816s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13412024-09-11T22:50:13.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 819.238676ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
13422024-09-11T22:50:13.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.015977459s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
13432024-09-11T22:50:13.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 814.013183ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13442024-09-11T22:50:13.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 945.951332ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
13452024-09-11T22:50:14.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 662.580596ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
13462024-09-11T22:50:14.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 972.254782ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
13472024-09-11T22:50:14.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 614.973597ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
13482024-09-11T22:50:14.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 749.217863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
13492024-09-11T22:50:14.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.220310511s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
13502024-09-11T22:50:14.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 620.836585ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
13512024-09-11T22:50:14.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.110533235s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13522024-09-11T22:50:14.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.305394071s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
13532024-09-11T22:50:14.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 515.968834ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
13542024-09-11T22:50:14.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 543.264848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
13552024-09-11T22:50:14.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 786.1364ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
13562024-09-11T22:50:14.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.161455857s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
13572024-09-11T22:50:14.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 945.954967ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
13582024-09-11T22:50:14.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 794.77102ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
13592024-09-11T22:50:14.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.168634899s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
13602024-09-11T22:50:14.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 767.2571ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
13612024-09-11T22:50:14.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 594.05412ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13622024-09-11T22:50:14.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 770.604354ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13632024-09-11T22:50:14.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 942.698206ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
13642024-09-11T22:50:14.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.385583905s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
13652024-09-11T22:50:14.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 749.472333ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
13662024-09-11T22:50:14.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 783.714323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
13672024-09-11T22:50:14.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 965.690195ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
13682024-09-11T22:50:15.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.246575367s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
13692024-09-11T22:50:15.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 1.070270441s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
13702024-09-11T22:50:15.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 595.18315ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
13712024-09-11T22:50:15.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 843.553533ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
13722024-09-11T22:50:15.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.256044471s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13732024-09-11T22:50:15.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.41462654s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
13742024-09-11T22:50:15.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 816.668382ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
13752024-09-11T22:50:15.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.28144746s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
13762024-09-11T22:50:15.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 949.497101ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13772024-09-11T22:50:15.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.060475978s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
13782024-09-11T22:50:15.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 715.083807ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
13792024-09-11T22:50:15.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 502.170776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
13802024-09-11T22:50:15.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.147166341s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
13812024-09-11T22:50:15.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.443493424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
13822024-09-11T22:50:15.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.456210569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
13832024-09-11T22:50:15.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 698.381875ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
13842024-09-11T22:50:15.808ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.018584813s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
13852024-09-11T22:50:15.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.409477604s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
13862024-09-11T22:50:15.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.478256147s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
13872024-09-11T22:50:16.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 1.101398069s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
13882024-09-11T22:50:16.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 592.131706ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
13892024-09-11T22:50:16.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.196614006s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
13902024-09-11T22:50:16.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 628.348177ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
13912024-09-11T22:50:16.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.341499733s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
13922024-09-11T22:50:16.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.115863499s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
13932024-09-11T22:50:16.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 930.901323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
13942024-09-11T22:50:16.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 817.566276ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
13952024-09-11T22:50:16.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.107892924s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
13962024-09-11T22:50:16.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 539.404908ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
13972024-09-11T22:50:16.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.193713481s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
13982024-09-11T22:50:16.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.31818311s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
13992024-09-11T22:50:16.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.082863465s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
14002024-09-11T22:50:16.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 853.87625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
14012024-09-11T22:50:16.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 979.644197ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
14022024-09-11T22:50:16.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 996.506285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
14032024-09-11T22:50:16.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 887.667107ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
14042024-09-11T22:50:17.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 762.058785ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
14052024-09-11T22:50:17.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 502.240705ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
14062024-09-11T22:50:17.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 935.709983ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
14072024-09-11T22:50:17.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 1.053679418s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
14082024-09-11T22:50:17.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 783.181438ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
14092024-09-11T22:50:17.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 736.591223ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
14102024-09-11T22:50:17.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.485465788s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
14112024-09-11T22:50:17.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.46454088s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
14122024-09-11T22:50:17.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 543.834896ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
14132024-09-11T22:50:17.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.151859946s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
14142024-09-11T22:50:17.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 770.31361ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
14152024-09-11T22:50:17.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.266286942s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
14162024-09-11T22:50:17.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.40590797s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
14172024-09-11T22:50:17.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.06975284s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
14182024-09-11T22:50:17.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.387521377s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
14192024-09-11T22:50:17.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 885.017394ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
14202024-09-11T22:50:17.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.347491023s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
14212024-09-11T22:50:17.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 925.979522ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
14222024-09-11T22:50:17.909ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.437240095s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
14232024-09-11T22:50:17.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.262684265s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
14242024-09-11T22:50:17.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.372516506s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
14252024-09-11T22:50:18.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.395732567s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
14262024-09-11T22:50:18.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 747.126583ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
14272024-09-11T22:50:18.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 931.17174ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
14282024-09-11T22:50:18.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.405452548s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
14292024-09-11T22:50:18.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 540.438172ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
14302024-09-11T22:50:18.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 530.430499ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
14312024-09-11T22:50:18.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 558.748791ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
14322024-09-11T22:50:18.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 547.555606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
14332024-09-11T22:50:18.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.308901682s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
14342024-09-11T22:50:18.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.16476096s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
14352024-09-11T22:50:18.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 851.348617ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
14362024-09-11T22:50:18.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.161141001s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
14372024-09-11T22:50:18.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 844.252802ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
14382024-09-11T22:50:18.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.271804168s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
14392024-09-11T22:50:18.937ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 636.948126ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
14402024-09-11T22:50:18.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 732.610556ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
14412024-09-11T22:50:18.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.243942872s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
14422024-09-11T22:50:19.113ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 607.885163ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
14432024-09-11T22:50:19.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.284600189s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
14442024-09-11T22:50:19.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 956.845716ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
14452024-09-11T22:50:19.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.490717694s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
14462024-09-11T22:50:19.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.047943822s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
14472024-09-11T22:50:19.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 705.504714ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
14482024-09-11T22:50:19.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.423687646s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
14492024-09-11T22:50:19.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 554.932148ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
14502024-09-11T22:50:19.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.270449188s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
14512024-09-11T22:50:19.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.286214319s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
14522024-09-11T22:50:19.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.193006896s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
14532024-09-11T22:50:19.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 749.161919ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
14542024-09-11T22:50:19.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 586.480714ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
14552024-09-11T22:50:19.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.196990718s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
14562024-09-11T22:50:19.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 505.179055ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
14572024-09-11T22:50:19.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 964.003097ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
14582024-09-11T22:50:19.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 915.012891ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
14592024-09-11T22:50:19.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 741.164426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
14602024-09-11T22:50:20.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.019484002s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
14612024-09-11T22:50:20.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.19086837s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
14622024-09-11T22:50:20.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.002347702s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
14632024-09-11T22:50:20.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 542.7591ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
14642024-09-11T22:50:20.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.164704891s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
14652024-09-11T22:50:20.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.030730152s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
14662024-09-11T22:50:20.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 585.843111ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
14672024-09-11T22:50:20.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.349180584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
14682024-09-11T22:50:20.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 654.953725ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
14692024-09-11T22:50:20.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 819.108622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
14702024-09-11T22:50:20.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.017339998s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
14712024-09-11T22:50:20.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 983.452096ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
14722024-09-11T22:50:20.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.178587044s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
14732024-09-11T22:50:20.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 541.991189ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
14742024-09-11T22:50:20.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.353340437s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
14752024-09-11T22:50:20.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.175409983s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
14762024-09-11T22:50:20.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 576.251516ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
14772024-09-11T22:50:20.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 554.279617ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
14782024-09-11T22:50:20.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.462927475s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
14792024-09-11T22:50:20.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.243979061s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
14802024-09-11T22:50:20.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 863.593205ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
14812024-09-11T22:50:21.064ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 563.604685ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
14822024-09-11T22:50:21.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 652.464319ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
14832024-09-11T22:50:21.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.061646327s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
14842024-09-11T22:50:21.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 944.409514ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
14852024-09-11T22:50:21.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 780.409737ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
14862024-09-11T22:50:21.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 761.752057ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
14872024-09-11T22:50:21.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.188489936s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
14882024-09-11T22:50:21.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 619.876356ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
14892024-09-11T22:50:21.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 858.986266ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
14902024-09-11T22:50:21.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.216656107s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
14912024-09-11T22:50:21.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.247441569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
14922024-09-11T22:50:21.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 964.158055ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
14932024-09-11T22:50:21.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 771.970112ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
14942024-09-11T22:50:21.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.362824593s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
14952024-09-11T22:50:21.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 544.283834ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
14962024-09-11T22:50:21.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.325534179s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
14972024-09-11T22:50:21.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 767.764006ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
14982024-09-11T22:50:21.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 921.586419ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
14992024-09-11T22:50:22.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 556.423378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
15002024-09-11T22:50:22.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 979.977719ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
15012024-09-11T22:50:22.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 917.0998ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
15022024-09-11T22:50:22.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 918.002467ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
15032024-09-11T22:50:22.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.177075398s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
15042024-09-11T22:50:22.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 791.621731ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
15052024-09-11T22:50:22.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 785.586252ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
15062024-09-11T22:50:22.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.099665222s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
15072024-09-11T22:50:22.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 614.97815ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
15082024-09-11T22:50:22.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.196865899s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
15092024-09-11T22:50:22.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.247051599s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
15102024-09-11T22:50:22.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.024715356s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
15112024-09-11T22:50:22.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 684.147669ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
15122024-09-11T22:50:22.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 872.871181ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
15132024-09-11T22:50:22.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.166820034s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
15142024-09-11T22:50:22.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.028433538s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
15152024-09-11T22:50:22.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 805.195771ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
15162024-09-11T22:50:22.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 915.259844ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
15172024-09-11T22:50:22.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 510.465258ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
15182024-09-11T22:50:23.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.338000513s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
15192024-09-11T22:50:23.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.072993766s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
15202024-09-11T22:50:23.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 864.67049ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
15212024-09-11T22:50:23.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 990.645869ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
15222024-09-11T22:50:23.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.073303661s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
15232024-09-11T22:50:23.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.263799374s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
15242024-09-11T22:50:23.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.477261066s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
15252024-09-11T22:50:23.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 754.73355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
15262024-09-11T22:50:23.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.212825708s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
15272024-09-11T22:50:23.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 531.808163ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
15282024-09-11T22:50:23.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 868.226025ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
15292024-09-11T22:50:23.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.199421883s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
15302024-09-11T22:50:23.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 655.441512ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
15312024-09-11T22:50:23.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 996.131624ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
15322024-09-11T22:50:23.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 753.713508ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
15332024-09-11T22:50:23.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 557.791818ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
15342024-09-11T22:50:23.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 808.812183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
15352024-09-11T22:50:23.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.216110993s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
15362024-09-11T22:50:23.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.117007644s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
15372024-09-11T22:50:23.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.053839329s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
15382024-09-11T22:50:23.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 858.971602ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
15392024-09-11T22:50:24.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 815.55214ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
15402024-09-11T22:50:24.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 509.114745ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
15412024-09-11T22:50:24.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.047286327s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
15422024-09-11T22:50:24.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.3987987s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
15432024-09-11T22:50:24.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 826.248625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
15442024-09-11T22:50:24.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0") failed: Property not found. retry in 1.099193315s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
15452024-09-11T22:50:24.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 748.753847ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
15462024-09-11T22:50:24.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 811.482968ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
15472024-09-11T22:50:24.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 656.773224ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
15482024-09-11T22:50:24.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.286671687s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
15492024-09-11T22:50:24.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.325974333s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
15502024-09-11T22:50:24.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 684.36389ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
15512024-09-11T22:50:24.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 850.76139ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
15522024-09-11T22:50:24.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.151679179s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
15532024-09-11T22:50:24.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.494806306s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
15542024-09-11T22:50:24.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 804.933986ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
15552024-09-11T22:50:24.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.484486409s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
15562024-09-11T22:50:24.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.334194423s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
15572024-09-11T22:50:24.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 923.54191ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
15582024-09-11T22:50:25.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 611.872333ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
15592024-09-11T22:50:25.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.10958713s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
15602024-09-11T22:50:25.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.48107751s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
15612024-09-11T22:50:25.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.229043437s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
15622024-09-11T22:50:25.113ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 750.15685ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
15632024-09-11T22:50:25.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 904.131093ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
15642024-09-11T22:50:25.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.317629133s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
15652024-09-11T22:50:25.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.496078465s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
15662024-09-11T22:50:25.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.35397321s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
15672024-09-11T22:50:25.485ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.11428066s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
15682024-09-11T22:50:25.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.006244296s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
15692024-09-11T22:50:25.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.165103025s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
15702024-09-11T22:50:25.593ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41865
15712024-09-11T22:50:25.595ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2134 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41865 req_id = bdd5bdf7-7395-4d78-bbba-b6ad8011fc85 response_code = 200 uri = /baseboard
15722024-09-11T22:50:25.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.317152875s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
15732024-09-11T22:50:25.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.06486349s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
15742024-09-11T22:50:25.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.015425363s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
15752024-09-11T22:50:25.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 1.316117896s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
15762024-09-11T22:50:25.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.132513784s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
15772024-09-11T22:50:25.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.294017121s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
15782024-09-11T22:50:26.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.406492311s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
15792024-09-11T22:50:26.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 688.364433ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
15802024-09-11T22:50:26.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 574.66425ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
15812024-09-11T22:50:26.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 675.599164ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
15822024-09-11T22:50:26.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.304842273s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
15832024-09-11T22:50:26.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.002045396s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
15842024-09-11T22:50:26.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 568.258849ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
15852024-09-11T22:50:26.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 829.354389ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
15862024-09-11T22:50:26.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.3657952s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
15872024-09-11T22:50:26.770ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 836.866057ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
15882024-09-11T22:50:26.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.27534758s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
15892024-09-11T22:50:26.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 887.293157ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
15902024-09-11T22:50:26.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.334504202s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
15912024-09-11T22:50:26.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 881.478366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
15922024-09-11T22:50:26.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 898.574059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
15932024-09-11T22:50:26.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 559.8359ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
15942024-09-11T22:50:26.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.392498399s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
15952024-09-11T22:50:26.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.499053031s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
15962024-09-11T22:50:27.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.412045929s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
15972024-09-11T22:50:27.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.196224783s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
15982024-09-11T22:50:27.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 972.563222ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
15992024-09-11T22:50:27.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 623.225341ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
16002024-09-11T22:50:27.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.204527888s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
16012024-09-11T22:50:27.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 584.117183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
16022024-09-11T22:50:27.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 1.280602559s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
16032024-09-11T22:50:27.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 793.224137ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
16042024-09-11T22:50:27.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 897.236611ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
16052024-09-11T22:50:27.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.145457861s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
16062024-09-11T22:50:27.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.004711476s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
16072024-09-11T22:50:27.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 783.086197ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
16082024-09-11T22:50:27.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 731.150498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
16092024-09-11T22:50:27.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.119264212s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
16102024-09-11T22:50:27.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.334279265s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
16112024-09-11T22:50:28.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.129010761s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
16122024-09-11T22:50:28.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.413778233s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
16132024-09-11T22:50:28.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 657.263568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
16142024-09-11T22:50:28.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 1.370300111s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
16152024-09-11T22:50:28.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 1.218986574s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
16162024-09-11T22:50:28.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.389854008s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
16172024-09-11T22:50:28.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.414768399s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
16182024-09-11T22:50:28.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 654.023894ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
16192024-09-11T22:50:28.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 948.421847ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
16202024-09-11T22:50:28.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56") failed: Property not found. retry in 1.494913199s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
16212024-09-11T22:50:28.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 1.239474561s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
16222024-09-11T22:50:28.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.397419061s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
16232024-09-11T22:50:28.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.435594933s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
16242024-09-11T22:50:28.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 828.136914ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
16252024-09-11T22:50:28.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0") failed: Property not found. retry in 666.609212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
16262024-09-11T22:50:28.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 512.043232ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
16272024-09-11T22:50:28.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244") failed: Property not found. retry in 1.286988255s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
16282024-09-11T22:50:29.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9") failed: Property not found. retry in 1.238450047s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
16292024-09-11T22:50:29.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.359541574s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
16302024-09-11T22:50:29.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 672.359136ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
16312024-09-11T22:50:29.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80") failed: Property not found. retry in 1.328272484s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
16322024-09-11T22:50:29.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 847.069374ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
16332024-09-11T22:50:29.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277") failed: Property not found. retry in 1.348520813s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
16342024-09-11T22:50:29.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f") failed: Property not found. retry in 1.480953171s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
16352024-09-11T22:50:29.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.282870955s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
16362024-09-11T22:50:29.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718") failed: Property not found. retry in 577.932535ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
16372024-09-11T22:50:29.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 633.222257ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
16382024-09-11T22:50:29.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d") failed: Property not found. retry in 1.377504519s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
16392024-09-11T22:50:29.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 895.491415ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
16402024-09-11T22:50:29.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 727.308986ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
16412024-09-11T22:50:29.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 616.080205ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
16422024-09-11T22:50:29.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82") failed: Property not found. retry in 1.497789629s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
16432024-09-11T22:50:30.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.372228033s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
16442024-09-11T22:50:30.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef") failed: Property not found. retry in 1.357617994s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
16452024-09-11T22:50:30.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 1.327503461s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
16462024-09-11T22:50:30.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 503.008056ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
16472024-09-11T22:50:30.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd") failed: Property not found. retry in 1.048358965s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
16482024-09-11T22:50:30.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.053159778s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
16492024-09-11T22:50:30.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 698.793771ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
16502024-09-11T22:50:30.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 692.646244ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
16512024-09-11T22:50:30.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.376246634s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
16522024-09-11T22:50:31.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 621.049997ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
16532024-09-11T22:50:31.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 502.187822ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
16542024-09-11T22:50:31.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62") failed: Property not found. retry in 1.382757208s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
16552024-09-11T22:50:31.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 778.8524ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
16562024-09-11T22:50:31.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f") failed: Property not found. retry in 1.439681138s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
16572024-09-11T22:50:31.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f") failed: Property not found. retry in 1.059264681s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
16582024-09-11T22:50:32.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88") failed: Property not found. retry in 725.032002ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
16592024-09-11T22:50:32.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf") failed: Property not found. retry in 869.012462ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
16602024-09-11T22:50:32.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b") failed: Property not found. retry in 1.045037476s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
16612024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_pantry_bbde0be4-b1a6-44b2-9c97-d73f27be6718
16622024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_75ac48f4-97b5-446c-bf76-d6d4c5e42f80
16632024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_b8d92285-3041-49be-9542-cbf9c8a2e3e0
16642024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_nexus_6a5513cb-d653-446c-8488-91cb1d7e4e8f
16652024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_oximeter_60c011c6-d26c-45f9-887c-70fa53b449b0
16662024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_9f252058-a33d-4f45-86c5-bc171de68d56
16672024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_nexus_88f8c4ef-0034-4c46-8187-a4bae0b36b62
16682024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_external_dns_c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf
16692024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_clickhouse_8c39300d-43ce-4e36-8a27-0fe8091f4f2f
16702024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_pantry_6366d8dc-d4d9-43fe-8e81-251bf7f28abd
16712024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_3c490ca9-6a8f-485c-b3a2-e1e8571725e9
16722024-09-11T22:50:33.445ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_5fbf461d-9a9b-46ff-acc7-244b7fd6b244
16732024-09-11T22:50:33.446ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_pantry_adffb264-4c4d-41de-9bc2-166e2d0d5e82
16742024-09-11T22:50:33.446ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_cc65a6a4-1064-4ddb-bfa9-3a644ccbc73d
16752024-09-11T22:50:33.446ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_external_dns_a9318a9c-1482-41a4-ba3e-a515b62e612b
16762024-09-11T22:50:33.446ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_nexus_098a841e-649e-4d26-b7e0-8ce27e09ad88
16772024-09-11T22:50:33.446ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_d36fee4e-6ff8-4647-a0bf-5944c7102e9f
16782024-09-11T22:50:33.446ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_abe48a0b-4221-441c-9356-29c0d26548ef
16792024-09-11T22:50:33.446ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3226 zone = oxz_crucible_64ce5226-a6ee-4705-8a1c-d6a6d9056277
16802024-09-11T22:50:33.447ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16812024-09-11T22:50:33.448ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16822024-09-11T22:50:33.461ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39331775 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52951 req_id = ac171596-63e4-4044-8466-31018861f577 response_code = 204 uri = /omicron-zones
16832024-09-11T22:50:33.466ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1322
16842024-09-11T22:50:33.466ZINFOSledAgent (RSS): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
16852024-09-11T22:50:33.466ZINFOSledAgent (RSS): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
16862024-09-11T22:50:33.470ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
16872024-09-11T22:50:33.470ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:720
16882024-09-11T22:50:33.507ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:887
16892024-09-11T22:50:33.509ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16902024-09-11T22:50:33.581ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16912024-09-11T22:50:33.750ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16922024-09-11T22:50:33.882ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16932024-09-11T22:50:34.068ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16942024-09-11T22:50:34.377ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16952024-09-11T22:50:34.553ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.164538498s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
16962024-09-11T22:50:34.717ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16972024-09-11T22:50:34.875ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16982024-09-11T22:50:35.169ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
16992024-09-11T22:50:35.785ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17002024-09-11T22:50:36.342ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17012024-09-11T22:50:37.267ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17022024-09-11T22:50:37.755ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 5.398418394s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
17032024-09-11T22:50:38.043ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17042024-09-11T22:50:39.169ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17052024-09-11T22:50:40.104ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17062024-09-11T22:50:41.524ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17072024-09-11T22:50:43.186ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 21.915706086s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
17082024-09-11T22:50:43.629ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17092024-09-11T22:50:45.402ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17102024-09-11T22:50:47.261ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17112024-09-11T22:50:50.296ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17122024-09-11T22:50:55.045ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/ad186af8-8c6c-4691-b5b3-63d42bf4113c/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
17132024-09-11T22:50:55.637ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36077
17142024-09-11T22:50:55.637ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36077 req_id = 2b97a0c0-f43a-407d-94b3-5e38c2fdd097 response_code = 200 uri = /baseboard
17152024-09-11T22:50:59.110ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "4004c0ec-0584-40ae-9c2a-54e7596e0657", "content-length": "173", "date": "Wed, 11 Sep 2024 22:50:58 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"ad186af8-8c6c-4691-b5b3-63d42bf4113c\"", request_id: "4004c0ec-0584-40ae-9c2a-54e7596e0657" } file = sled-agent/src/rack_setup/service.rs:949
17162024-09-11T22:51:03.252ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "fe5cbfcd-3ce7-4e48-a0da-c1b1f0a4ee98", "content-length": "173", "date": "Wed, 11 Sep 2024 22:51:02 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"35545f26-e1d9-4ad2-8ed3-679cba94411b\"", request_id: "fe5cbfcd-3ce7-4e48-a0da-c1b1f0a4ee98" } file = sled-agent/src/rack_setup/service.rs:949
17172024-09-11T22:51:05.145ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 33.659851228s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"fb902b09-d232-4697-9c1b-e9cdc12d9128\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 11 Sep 2024 22:51:05 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"fb902b09-d232-4697-9c1b-e9cdc12d9128\\" }" file = oximeter/producer/src/lib.rs:424 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
17182024-09-11T22:51:10.302ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "dca67985-e90e-47ed-8693-d59481199dfd", "content-length": "173", "date": "Wed, 11 Sep 2024 22:51:10 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"35545f26-e1d9-4ad2-8ed3-679cba94411b\"", request_id: "dca67985-e90e-47ed-8693-d59481199dfd" } file = sled-agent/src/rack_setup/service.rs:949
17192024-09-11T22:51:17.244ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
17202024-09-11T22:51:17.334ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60906
17212024-09-11T22:51:17.336ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17222024-09-11T22:51:17.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 190257 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60906 req_id = 69e132f4-0bdc-4095-b066-27523757d935 response_code = 200 uri = /inventory
17232024-09-11T22:51:17.525ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17242024-09-11T22:51:17.526ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
17252024-09-11T22:51:17.530ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
17262024-09-11T22:51:17.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6523 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60906 req_id = 9afb5e7d-dded-49a1-b786-2bcfd398c4cc response_code = 200 uri = /omicron-zones
17272024-09-11T22:51:22.562ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36870
17282024-09-11T22:51:22.564ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17292024-09-11T22:51:22.564ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17302024-09-11T22:51:22.564ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17312024-09-11T22:51:22.564ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17322024-09-11T22:51:22.564ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17332024-09-11T22:51:22.564ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17342024-09-11T22:51:22.564ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17352024-09-11T22:51:22.564ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2202 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36870 req_id = 4438c3eb-bcde-438d-ae03-5dbebb5d5a98 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17362024-09-11T22:51:22.566ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:959
17372024-09-11T22:51:23.143ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37461
17382024-09-11T22:51:23.144ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17392024-09-11T22:51:23.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 226105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37461 req_id = 6a26dd82-5bb1-456d-a93b-47151b85db94 response_code = 200 uri = /inventory
17402024-09-11T22:51:23.370ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17412024-09-11T22:51:23.370ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/config/omicron-zones.json file = common/src/ledger.rs:177
17422024-09-11T22:51:23.371ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/config/omicron-zones.json file = common/src/ledger.rs:177
17432024-09-11T22:51:23.371ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1515 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37461 req_id = 8283c704-d822-4468-b649-af51a4152454 response_code = 200 uri = /omicron-zones
17442024-09-11T22:51:23.519ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56531
17452024-09-11T22:51:23.519ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17462024-09-11T22:51:23.519ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17472024-09-11T22:51:23.519ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17482024-09-11T22:51:23.519ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17492024-09-11T22:51:23.519ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17502024-09-11T22:51:23.519ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17512024-09-11T22:51:23.519ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17522024-09-11T22:51:23.520ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56531 req_id = d91af5c1-f482-40e0-8383-47873c748da5 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17532024-09-11T22:51:23.595ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62855
17542024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17552024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17562024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17572024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17582024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17592024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17602024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17612024-09-11T22:51:23.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 226 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62855 req_id = 791eabad-fd09-49dd-b221-1db9fd34a723 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17622024-09-11T22:51:23.596ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55226
17632024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17642024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17652024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17662024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17672024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17682024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17692024-09-11T22:51:23.596ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17702024-09-11T22:51:23.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55226 req_id = 7fbfaf6e-d051-47a6-a3da-5a47057a2d72 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17712024-09-11T22:51:25.671ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41066
17722024-09-11T22:51:25.672ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 529 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41066 req_id = cd1b70ed-6b56-488d-be06-becf8d0623ac response_code = 200 uri = /baseboard
17732024-09-11T22:51:28.798ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41050
17742024-09-11T22:51:28.798ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41050 req_id = 201d2c00-3f01-4d10-a859-d7d1234cf4bf response_code = 200 uri = /vpc-routes
17752024-09-11T22:51:28.875ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60776
17762024-09-11T22:51:28.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60776 req_id = d3bc8808-4945-41dd-9d7a-3918cf687dc8 response_code = 200 uri = /v2p
17772024-09-11T22:51:28.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5646 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41050 req_id = ac0103f2-4e9c-46c9-9d84-5d0002ac00a7 response_code = 204 uri = /vpc-routes
17782024-09-11T22:51:29.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44441
17792024-09-11T22:51:29.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44441 req_id = 9017d7fe-eb0b-411b-bdb6-09d1c2f2b16a response_code = 200 uri = /vpc-routes
17802024-09-11T22:51:29.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44441 req_id = 4a9f2be3-cb5e-49f1-b122-169f6882e2e3 response_code = 204 uri = /vpc-routes
17812024-09-11T22:51:29.585ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39443
17822024-09-11T22:51:29.586ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 740 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39443 req_id = fa060b92-7d9e-4181-a514-f3d3591d8899 response_code = 200 uri = /vpc-routes
17832024-09-11T22:51:29.618ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50077
17842024-09-11T22:51:29.618ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50077 req_id = c1259abb-521e-4f49-9353-132e16d7e225 response_code = 200 uri = /v2p
17852024-09-11T22:51:29.668ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62480
17862024-09-11T22:51:29.680ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17872024-09-11T22:51:29.864ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17882024-09-11T22:51:29.978ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 310206 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:62480 req_id = 2de71e23-7d69-45bc-9d8a-33adc928282d response_code = 204 uri = /switch-ports
17892024-09-11T22:51:29.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57325
17902024-09-11T22:51:29.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64502
17912024-09-11T22:51:29.979ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17922024-09-11T22:51:29.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 875 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:39443 req_id = aa9d077f-23c0-4628-9eab-f8be8577cf91 response_code = 204 uri = /vpc-routes
17932024-09-11T22:51:30.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 33944 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62480 req_id = 29cc01d6-0baa-4f55-901a-5bfa8977e6e9 response_code = 200 uri = /network-bootstore-config
17942024-09-11T22:51:30.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43958
17952024-09-11T22:51:30.036ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 282 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43958 req_id = 507a8180-f2cf-487b-bbac-d856f7a6c2af response_code = 200 uri = /v2p
17962024-09-11T22:51:30.109ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
17972024-09-11T22:51:30.172ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
17982024-09-11T22:51:30.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 193094 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57325 req_id = 7b826fc1-d367-4535-9646-0802947c6300 response_code = 204 uri = /switch-ports
17992024-09-11T22:51:30.175ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 402 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57325 req_id = e033ffce-1882-4d44-b095-03e054315f7d response_code = 200 uri = /network-bootstore-config
18002024-09-11T22:51:30.210ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18012024-09-11T22:51:30.260ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
18022024-09-11T22:51:30.260ZINFOSledAgent (bootstore): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18032024-09-11T22:51:30.267ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 288072 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64502 req_id = d285b28d-ea4e-4480-ab4f-9d37d6a1f56b response_code = 204 uri = /switch-ports
18042024-09-11T22:51:30.270ZINFOSledAgent (bootstore): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18052024-09-11T22:51:30.283ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
18062024-09-11T22:51:30.283ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25239 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62480 req_id = 86fd9238-4604-4092-ba65-bda84b8810cf response_code = 204 uri = /network-bootstore-config
18072024-09-11T22:51:30.283ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 16079 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64502 req_id = 600ad2e5-3897-4043-a463-b193afb90d83 response_code = 200 uri = /network-bootstore-config
18082024-09-11T22:51:30.427ZINFOSledAgent (bootstore): Attempting to update network config with generation: 3, current_generation: 2 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
18092024-09-11T22:51:30.427ZINFOSledAgent (bootstore): Writing ledger to /pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18102024-09-11T22:51:30.428ZINFOSledAgent (bootstore): Writing ledger to /pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18112024-09-11T22:51:30.428ZINFOSledAgent (bootstore): Broadcasting network config with generation 3 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
18122024-09-11T22:51:30.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 515 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57325 req_id = 7c3ac40d-3d14-4142-a986-040928fb0e3b response_code = 204 uri = /network-bootstore-config
18132024-09-11T22:51:34.713ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58266
18142024-09-11T22:51:34.713ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58266 req_id = 924bdfba-15ac-4d03-9ff3-d4a3e5721369 response_code = 200 uri = /vpc-routes
18152024-09-11T22:51:34.720ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58266 req_id = 78f75c5f-5fd2-4dc4-a3e1-ce9277eb662d response_code = 204 uri = /vpc-routes
18162024-09-11T22:51:55.711ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:38508
18172024-09-11T22:51:55.712ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:38508 req_id = d8be1620-2171-4be6-9a02-72d6d89ef863 response_code = 200 uri = /baseboard
18182024-09-11T22:51:58.745ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44954
18192024-09-11T22:51:58.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44954 req_id = 1c226e5a-a40a-41f9-8fa7-d6ef9b0adeed response_code = 200 uri = /vpc-routes
18202024-09-11T22:51:58.769ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44954 req_id = ea9c928b-9dcc-479a-a5f8-ec376c67a26a response_code = 204 uri = /vpc-routes
18212024-09-11T22:51:58.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41232
18222024-09-11T22:51:58.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41232 req_id = 6fd9d597-c32f-4833-9f6c-57795596633e response_code = 200 uri = /v2p
18232024-09-11T22:51:59.290ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59136
18242024-09-11T22:51:59.291ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18252024-09-11T22:51:59.330ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18262024-09-11T22:51:59.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103586 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59136 req_id = 4394959c-6d20-4bb8-bf9e-00c4b3ab6dd5 response_code = 204 uri = /switch-ports
18272024-09-11T22:51:59.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2886 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59136 req_id = 449590d3-b449-4f1a-aad9-a4bbd4558851 response_code = 200 uri = /network-bootstore-config
18282024-09-11T22:51:59.669ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43059
18292024-09-11T22:51:59.669ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43059 req_id = 653b26ec-18c4-4882-9328-52fbc521de30 response_code = 200 uri = /vpc-routes
18302024-09-11T22:51:59.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43059 req_id = 13346d30-d1ce-4c22-a1e0-efb04f8394f7 response_code = 204 uri = /vpc-routes
18312024-09-11T22:51:59.741ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46628
18322024-09-11T22:51:59.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 841 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46628 req_id = 3e8bdc5a-8b6e-4483-b90a-22793efcb17a response_code = 200 uri = /vpc-routes
18332024-09-11T22:51:59.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 187 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46628 req_id = 2a3952cf-0c77-45e3-8e40-834d35a0e8bb response_code = 204 uri = /vpc-routes
18342024-09-11T22:51:59.893ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51384
18352024-09-11T22:51:59.894ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 927 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51384 req_id = 1b945778-3b31-493c-b8f5-45dfdbb7bcc5 response_code = 200 uri = /v2p
18362024-09-11T22:51:59.925ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44975
18372024-09-11T22:51:59.925ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44975 req_id = a891042b-d4be-4841-9f05-86681ea35fb5 response_code = 200 uri = /v2p
18382024-09-11T22:51:59.958ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63434
18392024-09-11T22:51:59.959ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18402024-09-11T22:51:59.995ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18412024-09-11T22:52:00.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99173 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63434 req_id = 71a2b250-a19a-4c1b-b3b7-e66002e774f5 response_code = 204 uri = /switch-ports
18422024-09-11T22:52:00.057ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33497
18432024-09-11T22:52:00.058ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18442024-09-11T22:52:00.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3828 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63434 req_id = 5baa624a-0152-44a3-976c-7d88cbd8071f response_code = 200 uri = /network-bootstore-config
18452024-09-11T22:52:00.094ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18462024-09-11T22:52:00.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94010 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:33497 req_id = 84060b3a-b896-45a2-a57c-fbf907727041 response_code = 204 uri = /switch-ports
18472024-09-11T22:52:00.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 886 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33497 req_id = c4c8a936-b60b-45c9-acc4-97147005c137 response_code = 200 uri = /network-bootstore-config
18482024-09-11T22:52:08.962ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:62558 remote_addr = [fd00:1122:3344:101::d]:36196 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
18492024-09-11T22:52:08.992ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 29400 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = b3961662-0819-4970-ae12-192e64ed617a response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
18502024-09-11T22:52:25.751ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53175
18512024-09-11T22:52:25.751ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 671 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53175 req_id = f0e46876-99cf-46ec-8056-09ed6500c351 response_code = 200 uri = /baseboard
18522024-09-11T22:52:28.876ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49066
18532024-09-11T22:52:28.877ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 700 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49066 req_id = 011cdab9-ea03-4b39-8775-2cc6261cac59 response_code = 200 uri = /vpc-routes
18542024-09-11T22:52:28.877ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62913
18552024-09-11T22:52:28.877ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62913 req_id = 0be28b8f-dd60-4a55-a8e5-4ece45a86b00 response_code = 200 uri = /v2p
18562024-09-11T22:52:29.014ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62227
18572024-09-11T22:52:29.015ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18582024-09-11T22:52:29.055ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18592024-09-11T22:52:29.114ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100080 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:62227 req_id = b66e0520-1887-4e65-9fcd-6a5713dde34d response_code = 204 uri = /switch-ports
18602024-09-11T22:52:29.114ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 172 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49066 req_id = 62cb7ec1-c7e1-4faf-a44a-ce369058e7a0 response_code = 204 uri = /vpc-routes
18612024-09-11T22:52:29.115ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 214 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62227 req_id = 046077cd-2d33-4303-a24f-490dfc0770c2 response_code = 200 uri = /network-bootstore-config
18622024-09-11T22:52:29.598ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60321
18632024-09-11T22:52:29.598ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60321 req_id = 5d1a9332-0f31-4bee-bf39-8acb2ec413e7 response_code = 200 uri = /vpc-routes
18642024-09-11T22:52:29.684ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43006
18652024-09-11T22:52:29.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43006 req_id = 38e8500f-4af4-4e3b-9ef4-1dabab9c4a44 response_code = 200 uri = /vpc-routes
18662024-09-11T22:52:29.697ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55103
18672024-09-11T22:52:29.697ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55103 req_id = 4e2eb919-7ee6-44e5-9e4d-1560ac360a80 response_code = 200 uri = /v2p
18682024-09-11T22:52:29.784ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43006 req_id = b2d8c665-5d4e-4623-86a5-86b85f81367d response_code = 204 uri = /vpc-routes
18692024-09-11T22:52:29.801ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41027
18702024-09-11T22:52:29.802ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18712024-09-11T22:52:29.843ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18722024-09-11T22:52:29.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102341 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:41027 req_id = b93b065f-5aa2-456b-afbc-61e49ee59754 response_code = 204 uri = /switch-ports
18732024-09-11T22:52:29.904ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62021
18742024-09-11T22:52:29.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60321 req_id = fb7cb035-195a-4221-aed9-734857477a7c response_code = 204 uri = /vpc-routes
18752024-09-11T22:52:29.905ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1020 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41027 req_id = 7be5764c-b545-4247-83fc-0fd18ff1bb4f response_code = 200 uri = /network-bootstore-config
18762024-09-11T22:52:29.905ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 993 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62021 req_id = 9236baae-ae59-42c3-9e3a-88de9798959d response_code = 200 uri = /v2p
18772024-09-11T22:52:29.975ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43876
18782024-09-11T22:52:29.976ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18792024-09-11T22:52:30.014ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18802024-09-11T22:52:30.071ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96016 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43876 req_id = 4f9b0cd0-2f92-4df7-93e9-2ba6b16b524c response_code = 204 uri = /switch-ports
18812024-09-11T22:52:30.072ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43876 req_id = 46de9393-8152-4521-821f-e3c082dc700a response_code = 200 uri = /network-bootstore-config
18822024-09-11T22:52:38.968ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6899 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = fcbf5156-6a56-41e5-8b08-7e61cb02264a response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
18832024-09-11T22:52:55.790ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48450
18842024-09-11T22:52:55.790ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48450 req_id = 4dadf060-ec53-4b72-a6c0-41786b33350f response_code = 200 uri = /baseboard
18852024-09-11T22:52:58.811ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40090
18862024-09-11T22:52:58.812ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40090 req_id = 44c266fe-ad69-45ed-aa21-e47a566b9534 response_code = 200 uri = /vpc-routes
18872024-09-11T22:52:58.837ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40090 req_id = 1907a0dc-0df1-4a68-ba05-e56796e41595 response_code = 204 uri = /vpc-routes
18882024-09-11T22:52:59.170ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52010
18892024-09-11T22:52:59.170ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52010 req_id = 925fae95-0396-4088-850f-eda9b014fd5a response_code = 200 uri = /v2p
18902024-09-11T22:52:59.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33818
18912024-09-11T22:52:59.255ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
18922024-09-11T22:52:59.296ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
18932024-09-11T22:52:59.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105742 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:33818 req_id = a8375efd-c5c8-4ece-afd7-7a2704f84c91 response_code = 204 uri = /switch-ports
18942024-09-11T22:52:59.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 235 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33818 req_id = 9c4ba9d1-b793-4264-ad71-e26c5c275b5a response_code = 200 uri = /network-bootstore-config
18952024-09-11T22:52:59.693ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41930
18962024-09-11T22:52:59.693ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41930 req_id = 292e18e0-a0d8-454d-ad23-488fdb744aeb response_code = 200 uri = /vpc-routes
18972024-09-11T22:52:59.707ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 837 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41930 req_id = ae546cb6-8007-497c-9757-a571bc3da117 response_code = 204 uri = /vpc-routes
18982024-09-11T22:52:59.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41044
18992024-09-11T22:52:59.748ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 867 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41044 req_id = b18a27f5-12b7-4b28-afe9-4bebd05e8090 response_code = 200 uri = /vpc-routes
19002024-09-11T22:52:59.759ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41044 req_id = 877ec398-f0f1-4be9-961a-bde4266b2199 response_code = 204 uri = /vpc-routes
19012024-09-11T22:52:59.861ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34889
19022024-09-11T22:52:59.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34889 req_id = 06b75680-908e-49d5-83f9-1e7e70f9ecf5 response_code = 200 uri = /v2p
19032024-09-11T22:52:59.909ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52243
19042024-09-11T22:52:59.909ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52243 req_id = 9a4edabf-0bfb-4f60-920d-caf9dde11ac8 response_code = 200 uri = /v2p
19052024-09-11T22:52:59.971ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41188
19062024-09-11T22:52:59.971ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19072024-09-11T22:53:00.013ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19082024-09-11T22:53:00.071ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100049 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:41188 req_id = 63b097b2-aa9f-4f97-8068-f1125381e448 response_code = 204 uri = /switch-ports
19092024-09-11T22:53:00.071ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52718
19102024-09-11T22:53:00.071ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19112024-09-11T22:53:00.104ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19122024-09-11T22:53:00.164ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92722 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52718 req_id = abf1acd8-3f3a-4278-a4eb-34d69b9a57df response_code = 204 uri = /switch-ports
19132024-09-11T22:53:00.164ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41188 req_id = 97ca1260-895c-41b5-8fb7-815ef40054c3 response_code = 200 uri = /network-bootstore-config
19142024-09-11T22:53:00.165ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 222 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52718 req_id = 0968d5d6-ad17-42bc-8833-dddda5e807fe response_code = 200 uri = /network-bootstore-config
19152024-09-11T22:53:08.968ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6364 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 5c6d0004-942b-41b6-a94d-949f30fb367c response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
19162024-09-11T22:53:25.825ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42177
19172024-09-11T22:53:25.825ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42177 req_id = 11c11f46-8e21-4468-940c-b7f832f245d4 response_code = 200 uri = /baseboard
19182024-09-11T22:53:28.878ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53000
19192024-09-11T22:53:28.878ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53000 req_id = 53f8215d-d417-46e6-9c98-2c144c8698e0 response_code = 200 uri = /vpc-routes
19202024-09-11T22:53:28.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53000 req_id = 1f8d1545-21a4-46c5-95b8-b3940c0d21e9 response_code = 204 uri = /vpc-routes
19212024-09-11T22:53:28.924ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65336
19222024-09-11T22:53:28.924ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 632 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65336 req_id = 3f25ef84-6eb0-4be9-a77d-e4078c87d3a5 response_code = 200 uri = /v2p
19232024-09-11T22:53:29.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36112
19242024-09-11T22:53:29.033ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19252024-09-11T22:53:29.075ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19262024-09-11T22:53:29.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100221 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36112 req_id = 0785a772-4655-47f1-9179-71128601b6d0 response_code = 204 uri = /switch-ports
19272024-09-11T22:53:29.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36112 req_id = 8c61910d-bd04-4976-953d-c6fd1c63c11e response_code = 200 uri = /network-bootstore-config
19282024-09-11T22:53:29.589ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54763
19292024-09-11T22:53:29.589ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54763 req_id = 5d86092a-9fca-4a10-af3f-6e00556f1e49 response_code = 200 uri = /vpc-routes
19302024-09-11T22:53:29.597ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54763 req_id = 85973b8c-133a-4ebb-bd72-fb4016721ee8 response_code = 204 uri = /vpc-routes
19312024-09-11T22:53:29.601ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53053
19322024-09-11T22:53:29.601ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53053 req_id = 7be721da-e5d3-4093-a0cf-59b71209ce8d response_code = 200 uri = /v2p
19332024-09-11T22:53:29.646ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41397
19342024-09-11T22:53:29.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 24 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41397 req_id = 815b28d0-6444-45e2-ae07-251323ea14c8 response_code = 200 uri = /vpc-routes
19352024-09-11T22:53:29.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41397 req_id = 42623a85-b055-4515-a5cb-24d377eec123 response_code = 204 uri = /vpc-routes
19362024-09-11T22:53:29.731ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37631
19372024-09-11T22:53:29.731ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19382024-09-11T22:53:29.774ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19392024-09-11T22:53:29.829ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98390 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37631 req_id = c4677cdf-179e-480a-b8e7-39d5d0823eb0 response_code = 204 uri = /switch-ports
19402024-09-11T22:53:29.830ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61721
19412024-09-11T22:53:29.830ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61536
19422024-09-11T22:53:29.830ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19432024-09-11T22:53:29.833ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3650 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37631 req_id = 2392b45d-a66e-444a-94dd-2d79acf07f51 response_code = 200 uri = /network-bootstore-config
19442024-09-11T22:53:29.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3709 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61536 req_id = e67d1105-c992-487c-8470-c3fc4129f37f response_code = 200 uri = /v2p
19452024-09-11T22:53:29.865ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19462024-09-11T22:53:29.926ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96183 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61721 req_id = 46daea82-931f-46e5-a644-4c7ec984680d response_code = 204 uri = /switch-ports
19472024-09-11T22:53:29.926ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61721 req_id = 8c4549fd-1848-46dd-9127-4fc29961535a response_code = 200 uri = /network-bootstore-config
19482024-09-11T22:53:38.969ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7196 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 4cd9ad08-957a-468a-8886-68b17479f74a response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
19492024-09-11T22:53:49.574ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
19502024-09-11T22:53:55.862ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61926
19512024-09-11T22:53:55.862ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61926 req_id = e284730f-6a35-4975-b146-188cd29bfe0b response_code = 200 uri = /baseboard
19522024-09-11T22:53:59.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40929
19532024-09-11T22:53:59.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40929 req_id = d387134b-1c58-447e-b781-ab745fcf596e response_code = 200 uri = /vpc-routes
19542024-09-11T22:53:59.090ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47289
19552024-09-11T22:53:59.090ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47289 req_id = 722a2615-1506-418d-bf20-19ee0076df97 response_code = 200 uri = /v2p
19562024-09-11T22:53:59.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40929 req_id = 0d9062e0-b29b-4a77-bef2-90e815af7056 response_code = 204 uri = /vpc-routes
19572024-09-11T22:53:59.259ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65253
19582024-09-11T22:53:59.259ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19592024-09-11T22:53:59.300ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19602024-09-11T22:53:59.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96894 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65253 req_id = 172030e5-a2e2-432f-8a0f-7db56e246715 response_code = 204 uri = /switch-ports
19612024-09-11T22:53:59.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65253 req_id = bc770951-cd1c-43ed-a495-1ff3bab80d83 response_code = 200 uri = /network-bootstore-config
19622024-09-11T22:53:59.598ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35432
19632024-09-11T22:53:59.599ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35432 req_id = 61562387-1a77-4e4e-b15b-7f850239be79 response_code = 200 uri = /vpc-routes
19642024-09-11T22:53:59.617ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35432 req_id = da407d48-480d-444d-95b6-f22aaedd9667 response_code = 204 uri = /vpc-routes
19652024-09-11T22:53:59.798ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62599
19662024-09-11T22:53:59.799ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 805 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62599 req_id = 781d87c5-645a-44c1-9e03-02739f249747 response_code = 200 uri = /vpc-routes
19672024-09-11T22:53:59.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62599 req_id = b03b680a-8c27-4568-8adc-e489fff5cc37 response_code = 204 uri = /vpc-routes
19682024-09-11T22:53:59.900ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41017
19692024-09-11T22:53:59.900ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41017 req_id = 449dc75a-f385-4b4f-8b2e-ab54983e57e2 response_code = 200 uri = /v2p
19702024-09-11T22:53:59.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47920
19712024-09-11T22:53:59.960ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47920 req_id = df75f0b1-be7c-41d6-8f43-47bd61944be3 response_code = 200 uri = /v2p
19722024-09-11T22:54:00.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64803
19732024-09-11T22:54:00.034ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19742024-09-11T22:54:00.075ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19752024-09-11T22:54:00.132ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98692 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64803 req_id = 4bf8939a-1760-4bac-9afa-bc3260e6335d response_code = 204 uri = /switch-ports
19762024-09-11T22:54:00.132ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46789
19772024-09-11T22:54:00.132ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19782024-09-11T22:54:00.165ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19792024-09-11T22:54:00.217ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85112 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46789 req_id = e93b0b88-11e2-4af3-bb4a-be1fe64a9331 response_code = 204 uri = /switch-ports
19802024-09-11T22:54:00.218ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64803 req_id = bba00371-2bcf-4c5b-8fcd-85c0b5444169 response_code = 200 uri = /network-bootstore-config
19812024-09-11T22:54:00.218ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46789 req_id = 1a43480d-0241-4bd5-9621-002d5f3cb8ef response_code = 200 uri = /network-bootstore-config
19822024-09-11T22:54:08.975ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5989 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 74e40ca0-e0fc-4551-8e55-5293aa7b6de8 response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
19832024-09-11T22:54:25.893ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37468
19842024-09-11T22:54:25.894ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37468 req_id = 283bc12b-1941-4ba3-8420-2f099f8878d0 response_code = 200 uri = /baseboard
19852024-09-11T22:54:28.871ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57772
19862024-09-11T22:54:28.872ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 852 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57772 req_id = 66b02eca-2f54-4e1a-b1ee-bd603a34df16 response_code = 200 uri = /v2p
19872024-09-11T22:54:28.984ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43908
19882024-09-11T22:54:28.985ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 701 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43908 req_id = 886fba15-bda6-4f48-bc82-05dc262fce30 response_code = 200 uri = /vpc-routes
19892024-09-11T22:54:28.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43908 req_id = fa7f2abf-7788-4347-9b0d-a6c88d38f216 response_code = 204 uri = /vpc-routes
19902024-09-11T22:54:29.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40146
19912024-09-11T22:54:29.033ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
19922024-09-11T22:54:29.072ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
19932024-09-11T22:54:29.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93949 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40146 req_id = 8e8bf402-389c-4c95-8d87-63a2fb269b6e response_code = 204 uri = /switch-ports
19942024-09-11T22:54:29.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40146 req_id = 959ec8ae-d3c9-4049-8339-0bfa01af38a8 response_code = 200 uri = /network-bootstore-config
19952024-09-11T22:54:29.627ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37124
19962024-09-11T22:54:29.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37124 req_id = 046e20a7-8f75-4d8d-ac5d-a71f951543bd response_code = 200 uri = /vpc-routes
19972024-09-11T22:54:29.630ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33766
19982024-09-11T22:54:29.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33766 req_id = e82125ae-f99d-4ff8-afb8-b5665ef7a94c response_code = 200 uri = /vpc-routes
19992024-09-11T22:54:29.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37124 req_id = 6e14aaa8-9987-482d-9d9e-f49bebdb7f79 response_code = 204 uri = /vpc-routes
20002024-09-11T22:54:29.640ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33766 req_id = 7b56d08f-9704-4852-9c4c-fbfaad81ac1e response_code = 204 uri = /vpc-routes
20012024-09-11T22:54:29.691ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53009
20022024-09-11T22:54:29.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53009 req_id = 1d45f1ba-7dd3-4ffb-97a0-5c5d22cc7828 response_code = 200 uri = /v2p
20032024-09-11T22:54:29.708ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53330
20042024-09-11T22:54:29.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53330 req_id = 3dea2460-bd87-47aa-9e7c-e2c6212bdbad response_code = 200 uri = /v2p
20052024-09-11T22:54:29.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32878
20062024-09-11T22:54:29.732ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20072024-09-11T22:54:29.773ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20082024-09-11T22:54:29.831ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99029 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:32878 req_id = c14fbe7a-9145-4f57-832f-8939ea7114a9 response_code = 204 uri = /switch-ports
20092024-09-11T22:54:29.831ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61790
20102024-09-11T22:54:29.832ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20112024-09-11T22:54:29.836ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4468 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32878 req_id = 10c80ceb-fcc0-4e94-82ab-f405e5de9aca response_code = 200 uri = /network-bootstore-config
20122024-09-11T22:54:29.870ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20132024-09-11T22:54:29.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97150 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61790 req_id = 45b24ac8-f1b2-4adb-8953-6d48bb6f63f4 response_code = 204 uri = /switch-ports
20142024-09-11T22:54:29.929ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61790 req_id = 11ab5b66-4a7a-435f-acf8-76547faa414f response_code = 200 uri = /network-bootstore-config
20152024-09-11T22:54:39.044ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6825 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = ad16d2c7-6a16-44bc-9e1e-f6f922093dd4 response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
20162024-09-11T22:54:55.927ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61853
20172024-09-11T22:54:55.928ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61853 req_id = 9de32091-c716-4234-8406-6aaed3db859f response_code = 200 uri = /baseboard
20182024-09-11T22:54:58.875ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63555
20192024-09-11T22:54:58.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 233 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63555 req_id = 2ecd8fc3-ed77-4e37-83e8-fe16a1e79c9e response_code = 200 uri = /v2p
20202024-09-11T22:54:59.076ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62156
20212024-09-11T22:54:59.076ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62156 req_id = a2dc36e7-4812-4fbd-8fae-3d11b79275cd response_code = 200 uri = /vpc-routes
20222024-09-11T22:54:59.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62156 req_id = 7b5c6ac9-b811-4f1e-8ef7-79ec3bee7b06 response_code = 204 uri = /vpc-routes
20232024-09-11T22:54:59.181ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57046
20242024-09-11T22:54:59.181ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20252024-09-11T22:54:59.223ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20262024-09-11T22:54:59.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101499 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57046 req_id = 55527aaa-828c-4116-9313-167385d01762 response_code = 204 uri = /switch-ports
20272024-09-11T22:54:59.286ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3870 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57046 req_id = 39ef450d-be68-4368-a734-e1c1076af568 response_code = 200 uri = /network-bootstore-config
20282024-09-11T22:54:59.601ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35771
20292024-09-11T22:54:59.601ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35771 req_id = a592afa9-a9f0-420c-900c-08a76485abe8 response_code = 200 uri = /vpc-routes
20302024-09-11T22:54:59.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35771 req_id = 7eb1ab3f-ad3b-4208-911c-7cfcdd1e1178 response_code = 204 uri = /vpc-routes
20312024-09-11T22:54:59.713ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63592
20322024-09-11T22:54:59.714ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1572 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63592 req_id = ed067a71-2c31-4fca-9ca2-4ba870a03cd3 response_code = 200 uri = /v2p
20332024-09-11T22:54:59.829ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45425
20342024-09-11T22:54:59.830ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 971 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45425 req_id = 732c2071-4056-4950-997b-bfd797e10cfe response_code = 200 uri = /vpc-routes
20352024-09-11T22:54:59.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45425 req_id = 5dbd8d1a-deca-4572-a210-44bd14b7f9d2 response_code = 204 uri = /vpc-routes
20362024-09-11T22:54:59.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53581
20372024-09-11T22:54:59.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53581 req_id = 2051a2c1-0944-4ce7-aea4-5e984c5b9a0e response_code = 200 uri = /v2p
20382024-09-11T22:54:59.914ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52331
20392024-09-11T22:54:59.915ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20402024-09-11T22:54:59.953ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20412024-09-11T22:55:00.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95832 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52331 req_id = 49e88d94-7d73-4da2-8ada-618502bc3597 response_code = 204 uri = /switch-ports
20422024-09-11T22:55:00.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52331 req_id = 31dda447-8f6c-4e63-aa33-9a574585769a response_code = 200 uri = /network-bootstore-config
20432024-09-11T22:55:00.068ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47459
20442024-09-11T22:55:00.068ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20452024-09-11T22:55:00.102ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20462024-09-11T22:55:00.165ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97634 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47459 req_id = bcbe8fb4-f3da-4c61-8976-e15615afbac9 response_code = 204 uri = /switch-ports
20472024-09-11T22:55:00.166ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47459 req_id = b301dbc6-cd40-40ba-8862-57bff06917c2 response_code = 200 uri = /network-bootstore-config
20482024-09-11T22:55:09.044ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7376 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = ce58244b-23ed-46f2-a988-a1c04673f70b response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
20492024-09-11T22:55:25.960ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58161
20502024-09-11T22:55:25.960ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 134 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58161 req_id = c98cf297-6a70-4fb4-a333-a47f582e364d response_code = 200 uri = /baseboard
20512024-09-11T22:55:28.811ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51437
20522024-09-11T22:55:28.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 639 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51437 req_id = a381edb9-a5ee-47af-b703-e9b87c13f3df response_code = 200 uri = /vpc-routes
20532024-09-11T22:55:28.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51437 req_id = 1f31ce0b-124b-457f-b0b9-edfceaee86bd response_code = 204 uri = /vpc-routes
20542024-09-11T22:55:28.995ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61016
20552024-09-11T22:55:28.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61016 req_id = 4833653b-a500-435e-a352-31c3b664125b response_code = 200 uri = /v2p
20562024-09-11T22:55:29.013ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53176
20572024-09-11T22:55:29.013ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20582024-09-11T22:55:29.055ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20592024-09-11T22:55:29.113ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99998 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53176 req_id = b01132db-b70e-4b87-82f7-8702f4f1d9fb response_code = 204 uri = /switch-ports
20602024-09-11T22:55:29.113ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53176 req_id = b780784a-a1b9-4255-b0dc-81cfa3487041 response_code = 200 uri = /network-bootstore-config
20612024-09-11T22:55:29.603ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63712
20622024-09-11T22:55:29.604ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 897 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63712 req_id = 2ebf312a-dff7-421a-890a-29113270caef response_code = 200 uri = /vpc-routes
20632024-09-11T22:55:29.658ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41062
20642024-09-11T22:55:29.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41062 req_id = 43bc9776-0123-484d-af53-c64347fc0c88 response_code = 200 uri = /v2p
20652024-09-11T22:55:29.664ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63712 req_id = 55b29f79-a1c6-48bf-bbe2-2748a4266cf5 response_code = 204 uri = /vpc-routes
20662024-09-11T22:55:29.675ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46541
20672024-09-11T22:55:29.675ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 15 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46541 req_id = 24b58d32-c7df-41df-b8c5-c0791bb97443 response_code = 200 uri = /vpc-routes
20682024-09-11T22:55:29.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46541 req_id = ab5d66e4-aecc-4e4f-b245-76c69a744100 response_code = 204 uri = /vpc-routes
20692024-09-11T22:55:29.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42131
20702024-09-11T22:55:29.748ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20712024-09-11T22:55:29.788ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20722024-09-11T22:55:29.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100154 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42131 req_id = 1ca08388-ae50-4b1e-b300-bbfc26b71822 response_code = 204 uri = /switch-ports
20732024-09-11T22:55:29.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38859
20742024-09-11T22:55:29.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38704
20752024-09-11T22:55:29.848ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20762024-09-11T22:55:29.852ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4281 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38704 req_id = 3947dc41-05fa-4764-8977-feb50e334f15 response_code = 200 uri = /v2p
20772024-09-11T22:55:29.852ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4489 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42131 req_id = 28af2f61-4a16-41d9-9128-3981090d3f0e response_code = 200 uri = /network-bootstore-config
20782024-09-11T22:55:29.884ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20792024-09-11T22:55:29.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90785 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:38859 req_id = 19be6337-cdc2-44c1-8b86-bfc394f2f6ca response_code = 204 uri = /switch-ports
20802024-09-11T22:55:29.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38859 req_id = 685fe798-6cd5-4476-b13d-0bf7ed2ac983 response_code = 200 uri = /network-bootstore-config
20812024-09-11T22:55:39.044ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6925 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 2475fac1-a67c-4a8f-a946-bec74093b532 response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
20822024-09-11T22:55:55.993ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45072
20832024-09-11T22:55:55.993ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45072 req_id = 6767b768-f4cc-470c-934a-25aa473142aa response_code = 200 uri = /baseboard
20842024-09-11T22:55:58.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63258
20852024-09-11T22:55:58.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63258 req_id = dd55c19a-3648-4321-9b9c-d6738c9eb930 response_code = 200 uri = /vpc-routes
20862024-09-11T22:55:58.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63258 req_id = 3bda5637-f30f-4197-a966-9a989a934af8 response_code = 204 uri = /vpc-routes
20872024-09-11T22:55:59.193ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63872
20882024-09-11T22:55:59.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63872 req_id = df89db38-954a-44a9-bb72-6a774193865c response_code = 200 uri = /v2p
20892024-09-11T22:55:59.205ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52012
20902024-09-11T22:55:59.205ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
20912024-09-11T22:55:59.242ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
20922024-09-11T22:55:59.306ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101369 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52012 req_id = dfe83c79-c829-4b87-ae37-bbfd3cf4bc76 response_code = 204 uri = /switch-ports
20932024-09-11T22:55:59.306ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52012 req_id = ad227bc6-a967-460c-880b-6bae6dd79c10 response_code = 200 uri = /network-bootstore-config
20942024-09-11T22:55:59.391ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33433
20952024-09-11T22:55:59.392ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
20962024-09-11T22:55:59.392ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20972024-09-11T22:55:59.392ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
20982024-09-11T22:55:59.392ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20992024-09-11T22:55:59.392ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21002024-09-11T22:55:59.392ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21012024-09-11T22:55:59.392ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21022024-09-11T22:55:59.392ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 507 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33433 req_id = a4282b40-0866-4bca-bfe1-85f62b5f36ea response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
21032024-09-11T22:55:59.582ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55329
21042024-09-11T22:55:59.582ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55329 req_id = 6fde13ad-7492-4055-92f8-a631c16cb948 response_code = 200 uri = /vpc-routes
21052024-09-11T22:55:59.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 258 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55329 req_id = 7ea6029f-7a12-4c1e-99a2-8cceb701b79a response_code = 204 uri = /vpc-routes
21062024-09-11T22:55:59.689ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40541
21072024-09-11T22:55:59.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40541 req_id = 905beb7a-7764-4762-b6f8-8d7d3a07fbc6 response_code = 200 uri = /vpc-routes
21082024-09-11T22:55:59.834ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59362
21092024-09-11T22:55:59.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 156 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59362 req_id = d4971215-28c9-4a65-911a-ce1af0877847 response_code = 200 uri = /v2p
21102024-09-11T22:55:59.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40541 req_id = d5f15ce4-01be-4837-af75-48f9422d481c response_code = 204 uri = /vpc-routes
21112024-09-11T22:55:59.891ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42347
21122024-09-11T22:55:59.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42347 req_id = 30c4474b-f357-4758-bf8a-6f6afa300507 response_code = 200 uri = /v2p
21132024-09-11T22:55:59.954ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56208
21142024-09-11T22:55:59.954ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
21152024-09-11T22:55:59.993ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
21162024-09-11T22:56:00.052ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98256 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56208 req_id = 5eaeb015-81da-4a1d-a17b-6604ae1bfdd1 response_code = 204 uri = /switch-ports
21172024-09-11T22:56:00.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 668 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56208 req_id = 4bd42408-1a8b-4cfe-9f91-1c46d61c760e response_code = 200 uri = /network-bootstore-config
21182024-09-11T22:56:00.066ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52953
21192024-09-11T22:56:00.067ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
21202024-09-11T22:56:00.067ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21212024-09-11T22:56:00.067ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
21222024-09-11T22:56:00.067ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21232024-09-11T22:56:00.067ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21242024-09-11T22:56:00.067ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21252024-09-11T22:56:00.067ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21262024-09-11T22:56:00.067ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 440 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52953 req_id = d5fde76a-a5c1-4d08-a735-dcac8af03193 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
21272024-09-11T22:56:00.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45874
21282024-09-11T22:56:00.102ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
21292024-09-11T22:56:00.141ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
21302024-09-11T22:56:00.195ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92612 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45874 req_id = aa49925e-666e-48f4-a32c-a4356d1cc7c9 response_code = 204 uri = /switch-ports
21312024-09-11T22:56:00.195ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55991
21322024-09-11T22:56:00.195ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 248 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45874 req_id = 7e901c28-5150-4da5-a389-0447f5a60e61 response_code = 200 uri = /network-bootstore-config
21332024-09-11T22:56:00.195ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bd5846a9-8c6b-4c09-974b-f1d5f3391f63, kind: Service { id: c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf }, name: Name("external-dns-c0e4ec3d-62c9-496c-927b-3c7d51ac0eaf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 23])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d69533f1-7b03-41af-bd25-e4cac6e97a61, kind: Service { id: a9318a9c-1482-41a4-ba3e-a515b62e612b }, name: Name("external-dns-a9318a9c-1482-41a4-ba3e-a515b62e612b"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 188])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3d64456b-f2a4-4471-b30f-a388f7917506, kind: Service { id: 098a841e-649e-4d26-b7e0-8ce27e09ad88 }, name: Name("nexus-098a841e-649e-4d26-b7e0-8ce27e09ad88"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f4a8407-22ed-44fa-b5f2-82078e2b1752, kind: Service { id: 88f8c4ef-0034-4c46-8187-a4bae0b36b62 }, name: Name("nexus-88f8c4ef-0034-4c46-8187-a4bae0b36b62"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 229])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948186f9-0b1c-4f2f-8d1a-8d2e006f7dbe, kind: Service { id: 6a5513cb-d653-446c-8488-91cb1d7e4e8f }, name: Name("nexus-6a5513cb-d653-446c-8488-91cb1d7e4e8f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 82])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
21342024-09-11T22:56:00.195ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21352024-09-11T22:56:00.195ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
21362024-09-11T22:56:00.195ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21372024-09-11T22:56:00.195ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21382024-09-11T22:56:00.195ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21392024-09-11T22:56:00.196ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21402024-09-11T22:56:00.196ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 406 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55991 req_id = eb203fd9-8999-4d82-88b4-39ee3fd09ec6 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
21412024-09-11T22:56:09.046ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8793 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 6b89ee38-f385-4414-bdab-be21e776fe10 response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
21422024-09-11T22:56:26.030ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:62862
21432024-09-11T22:56:26.031ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:62862 req_id = 19b13d1a-d7f5-4af7-a1f2-f5de6f87736e response_code = 200 uri = /baseboard
21442024-09-11T22:56:28.854ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36501
21452024-09-11T22:56:28.854ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36501 req_id = a7964d27-683a-45b8-8c06-489ed2157d15 response_code = 200 uri = /vpc-routes
21462024-09-11T22:56:28.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 869 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36501 req_id = 8c42e1bc-ceb0-4ed7-8ba1-c3d7e6a83b26 response_code = 204 uri = /vpc-routes
21472024-09-11T22:56:28.971ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49644
21482024-09-11T22:56:28.971ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49644 req_id = b33a7d1e-8ceb-4f83-8a40-a6799db37ef5 response_code = 200 uri = /v2p
21492024-09-11T22:56:28.986ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36618
21502024-09-11T22:56:28.986ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
21512024-09-11T22:56:29.026ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
21522024-09-11T22:56:29.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95547 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36618 req_id = fd1c68f0-ecde-4520-a442-cc24b44e36f3 response_code = 204 uri = /switch-ports
21532024-09-11T22:56:29.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36618 req_id = 68c370b9-1f1c-4034-bec1-4e74ad5d6fba response_code = 200 uri = /network-bootstore-config
21542024-09-11T22:56:29.642ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62613
21552024-09-11T22:56:29.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62613 req_id = 56318bf0-0cb5-4278-8d43-fc1aa54fb023 response_code = 200 uri = /vpc-routes
21562024-09-11T22:56:29.652ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62613 req_id = 2722184e-9278-448f-9199-962b4576dcb5 response_code = 204 uri = /vpc-routes
21572024-09-11T22:56:29.726ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35998
21582024-09-11T22:56:29.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 676 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35998 req_id = e4ef7e67-d881-42f3-b691-e1891c9999b5 response_code = 200 uri = /v2p
21592024-09-11T22:56:29.740ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62897
21602024-09-11T22:56:29.741ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
21612024-09-11T22:56:29.783ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
21622024-09-11T22:56:29.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102656 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62897 req_id = 7c000f20-3d53-4879-aac9-e93ac42d9f53 response_code = 204 uri = /switch-ports
21632024-09-11T22:56:29.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60893
21642024-09-11T22:56:29.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51513
21652024-09-11T22:56:29.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60893 req_id = b67e8198-8232-4d28-9f11-f37876ab2129 response_code = 200 uri = /vpc-routes
21662024-09-11T22:56:29.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 227 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62897 req_id = bba76f45-d3e5-47b4-934b-cf0319761431 response_code = 200 uri = /network-bootstore-config
21672024-09-11T22:56:29.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 262 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51513 req_id = a2309b8a-894b-45ee-95d3-e259b675d8b1 response_code = 200 uri = /v2p
21682024-09-11T22:56:29.852ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60893 req_id = 11f0052a-0c1a-4ca4-a44b-ca4d6ad6a051 response_code = 204 uri = /vpc-routes
21692024-09-11T22:56:29.928ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44079
21702024-09-11T22:56:29.928ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
21712024-09-11T22:56:29.969ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
21722024-09-11T22:56:30.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103123 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44079 req_id = 70d49ea6-f4ef-4611-873a-b40f2f1190e1 response_code = 204 uri = /switch-ports
21732024-09-11T22:56:30.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44079 req_id = ce5292c2-c19f-49ca-ab66-6241a427aa2b response_code = 200 uri = /network-bootstore-config
21742024-09-11T22:56:39.045ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6740 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 35c51d6c-1952-4e47-832e-6fe710b8446f response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
21752024-09-11T22:56:56.064ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:39321
21762024-09-11T22:56:56.065ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 706 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:39321 req_id = aa71be84-2f90-4c88-83f3-8ad242691634 response_code = 200 uri = /baseboard
21772024-09-11T22:56:58.948ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34905
21782024-09-11T22:56:58.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34905 req_id = f58d6c9c-8c38-43c0-aa0f-b51671530226 response_code = 200 uri = /vpc-routes
21792024-09-11T22:56:58.958ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34905 req_id = 24c1c334-b417-455b-955a-8a7b188136cf response_code = 204 uri = /vpc-routes
21802024-09-11T22:56:59.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38105
21812024-09-11T22:56:59.101ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38105 req_id = 8e43b48f-f7a3-4aab-af6e-3bd1b5ee809a response_code = 200 uri = /v2p
21822024-09-11T22:56:59.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37986
21832024-09-11T22:56:59.139ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
21842024-09-11T22:56:59.179ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
21852024-09-11T22:56:59.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95239 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:37986 req_id = 0f4e707e-46d0-4b3e-9098-0887bd55f8a8 response_code = 204 uri = /switch-ports
21862024-09-11T22:56:59.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37986 req_id = 0960be35-8a58-49e9-bb96-37cf4d4bebc0 response_code = 200 uri = /network-bootstore-config
21872024-09-11T22:56:59.617ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54987
21882024-09-11T22:56:59.617ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54987 req_id = f623f247-f7f2-4332-94ff-ae188706a91a response_code = 200 uri = /vpc-routes
21892024-09-11T22:56:59.644ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54987 req_id = 4d44d52a-ca82-4c12-9488-6046f9b31c2e response_code = 204 uri = /vpc-routes
21902024-09-11T22:56:59.783ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58358
21912024-09-11T22:56:59.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58358 req_id = 00a1815b-96a9-456d-af3e-64b70bd328a4 response_code = 200 uri = /vpc-routes
21922024-09-11T22:56:59.784ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58986
21932024-09-11T22:56:59.785ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58986 req_id = c97e1903-a937-4b9b-8501-8d072b2acc7a response_code = 200 uri = /v2p
21942024-09-11T22:56:59.792ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58358 req_id = bbc33bcc-7cab-474a-b6d6-fe60b8b39c7a response_code = 204 uri = /vpc-routes
21952024-09-11T22:56:59.841ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36632
21962024-09-11T22:56:59.842ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
21972024-09-11T22:56:59.881ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
21982024-09-11T22:56:59.937ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95494 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36632 req_id = 4e59f37c-9d35-48f6-9af6-bb425a14eb88 response_code = 204 uri = /switch-ports
21992024-09-11T22:56:59.937ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62623
22002024-09-11T22:56:59.937ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46676
22012024-09-11T22:56:59.937ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 262 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36632 req_id = 7a50ab69-58b5-4036-88df-1f29b94849a3 response_code = 200 uri = /network-bootstore-config
22022024-09-11T22:56:59.937ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 259 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62623 req_id = 74ac25d5-9e82-4a08-a15e-8393f0192dfc response_code = 200 uri = /v2p
22032024-09-11T22:56:59.938ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
22042024-09-11T22:56:59.975ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
22052024-09-11T22:57:00.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94367 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46676 req_id = 42e2c408-7e2a-4200-8eb6-4139919db409 response_code = 204 uri = /switch-ports
22062024-09-11T22:57:00.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46676 req_id = 202af16d-44d0-4a85-bdc5-d4487a21a17e response_code = 200 uri = /network-bootstore-config
22072024-09-11T22:57:09.158ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6725 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 7411bcb7-5edc-4fc7-8603-2df983ba6d89 response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
22082024-09-11T22:57:25.702ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48785
22092024-09-11T22:57:25.702ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48785 req_id = d484634e-cb32-478b-908a-bde015b18d49 response_code = 200 uri = /v2p
22102024-09-11T22:57:25.703ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(4856321) }
22112024-09-11T22:57:25.703ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48785 req_id = 9907cc37-1b16-4e9d-9a6a-c574f4976839 response_code = 204 uri = /v2p
22122024-09-11T22:57:25.948ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45659
22132024-09-11T22:57:25.954ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:570 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(4856321))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-b5b9a533d8c", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 7ddb4576-a683-4e7c-86e7-612fc6b65676, read_only_parent: Some(Volume { block_size: 512, id: 403434d3-9cc5-4eec-98eb-e6d906edc387, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: c3796d9e-9fd7-4cf2-907c-eec5967ddf1c, key: Some("qPokywCavqiEWTyyCAMJQ6Zpb8tZK/sdG3FrIRaXx2I="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 7ddb4576-a683-4e7c-86e7-612fc6b65676, key: Some("3RwZN2Xpcynuwjjoe2HqUDTYRhPmS80MIVqOq1KfBk0="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::15]:19001", "[fd00:1122:3344:101::19]:19000"] } }] } }], cloud_init_bytes: Some(..) } instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed instance_runtime = InstanceRuntimeState { propolis_id: Some(4eb715b5-913d-40f1-9d20-812581b68eeb (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-09-11T22:57:25.394677550Z } metadata = InstanceMetadata { silo_id: ba5d3585-7489-44bd-96a9-660c1014b181, project_id: d76d2844-f07a-4f34-a551-157fbcb729a1 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-11T22:57:25.947936104Z }
22142024-09-11T22:57:25.954ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:603 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22152024-09-11T22:57:25.955ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1008 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed migration_id = None propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(4856321))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 979d808f-1b7f-4fb7-a785-ae62c151a740, kind: Instance { id: 8375b88a-00e2-48ff-979f-45fa96db29ed }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 163, 246])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4856321), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-b5b9a533d8c", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 7ddb4576-a683-4e7c-86e7-612fc6b65676, read_only_parent: Some(Volume { block_size: 512, id: 403434d3-9cc5-4eec-98eb-e6d906edc387, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: c3796d9e-9fd7-4cf2-907c-eec5967ddf1c, key: Some("qPokywCavqiEWTyyCAMJQ6Zpb8tZK/sdG3FrIRaXx2I="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 7ddb4576-a683-4e7c-86e7-612fc6b65676, key: Some("3RwZN2Xpcynuwjjoe2HqUDTYRhPmS80MIVqOq1KfBk0="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::15]:19001", "[fd00:1122:3344:101::19]:19000"] } }] } }], cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-11T22:57:25.947936104Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
22162024-09-11T22:57:25.956ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8204 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45659 req_id = e765aa42-aa98-4bb3-acee-1c9d1eede488 response_code = 200 uri = /vmms/4eb715b5-913d-40f1-9d20-812581b68eeb
22172024-09-11T22:57:26.066ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62624
22182024-09-11T22:57:26.067ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62624 req_id = 9a27d092-8546-408d-a8a5-aad3734f5105 response_code = 200 uri = /vmms/4eb715b5-913d-40f1-9d20-812581b68eeb/state
22192024-09-11T22:57:26.068ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 244, 163, 246]), slot: 0, vni: Vni { inner: 4856321 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
22202024-09-11T22:57:26.068ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22212024-09-11T22:57:26.101ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36176
22222024-09-11T22:57:26.102ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36176 req_id = c50381d4-556f-48ad-bccc-fff791d0fab6 response_code = 200 uri = /baseboard
22232024-09-11T22:57:26.109ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb file = illumos-utils/src/zone.rs:316 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22242024-09-11T22:57:26.131ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb file = illumos-utils/src/zone.rs:354 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22252024-09-11T22:57:27.246ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22262024-09-11T22:57:27.246ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb zone file = illumos-utils/src/running_zone.rs:488 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22272024-09-11T22:57:27.419ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 66.40254ms file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22282024-09-11T22:57:27.501ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 70.888135ms file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22292024-09-11T22:57:27.590ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 186.199151ms file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22302024-09-11T22:57:27.794ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 279.932022ms file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22312024-09-11T22:57:28.090ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 575.51196ms file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22322024-09-11T22:57:28.678ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 1.276537846s file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22332024-09-11T22:57:28.904ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44802
22342024-09-11T22:57:28.906ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46533
22352024-09-11T22:57:28.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46533 req_id = 1758ea85-048e-43ad-8ceb-2fcfa1f96e26 response_code = 200 uri = /vpc-routes
22362024-09-11T22:57:28.963ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46533 req_id = 7df7a292-92b7-4e38-805b-d9d43a518f8a response_code = 204 uri = /vpc-routes
22372024-09-11T22:57:29.231ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37354
22382024-09-11T22:57:29.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 181 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37354 req_id = c6f3543f-d0af-4c19-aaa4-5c8e2c65d51d response_code = 200 uri = /v2p
22392024-09-11T22:57:29.285ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56604
22402024-09-11T22:57:29.285ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
22412024-09-11T22:57:29.323ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
22422024-09-11T22:57:29.392ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106899 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:56604 req_id = 93c6b164-1498-4edc-80c9-a3fee0d3ee11 response_code = 204 uri = /switch-ports
22432024-09-11T22:57:29.392ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56604 req_id = cddf5a97-75c2-499e-963c-3011a8778c6a response_code = 200 uri = /network-bootstore-config
22442024-09-11T22:57:29.592ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34553
22452024-09-11T22:57:29.667ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48488
22462024-09-11T22:57:29.667ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48488 req_id = afbe3d3d-bd8e-47f4-bc2a-51e7153cffbd response_code = 200 uri = /vpc-routes
22472024-09-11T22:57:29.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43568
22482024-09-11T22:57:29.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43568 req_id = bcfc5729-b445-4e00-b816-93bff2afccaf response_code = 200 uri = /vpc-routes
22492024-09-11T22:57:29.707ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63015
22502024-09-11T22:57:29.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48488 req_id = 4de61693-28f7-41fa-bd67-1b97ea4ee2c3 response_code = 204 uri = /vpc-routes
22512024-09-11T22:57:29.736ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61983
22522024-09-11T22:57:29.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61983 req_id = cca79034-364d-4276-95e3-64aafd575cca response_code = 200 uri = /v2p
22532024-09-11T22:57:29.743ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43568 req_id = 38cf4cda-dec3-4427-8262-f0989c555341 response_code = 204 uri = /vpc-routes
22542024-09-11T22:57:29.815ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60507
22552024-09-11T22:57:29.815ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
22562024-09-11T22:57:29.855ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
22572024-09-11T22:57:29.912ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96527 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60507 req_id = bf67f7a8-d522-4b3e-b198-75b551e2ca36 response_code = 204 uri = /switch-ports
22582024-09-11T22:57:29.912ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37701
22592024-09-11T22:57:29.912ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63065
22602024-09-11T22:57:29.913ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4014
22612024-09-11T22:57:29.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4342 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60507 req_id = a56d38c1-bf40-426b-8076-5190c1c63675 response_code = 200 uri = /network-bootstore-config
22622024-09-11T22:57:29.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63065 req_id = 601fc5e3-e978-4167-bebb-afce2eec9979 response_code = 200 uri = /v2p
22632024-09-11T22:57:29.947ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4038
22642024-09-11T22:57:29.966ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 1.193159726s file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22652024-09-11T22:57:30.003ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90988 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37701 req_id = 0b9025b8-2b43-4a90-b5fb-241b932f1451 response_code = 204 uri = /switch-ports
22662024-09-11T22:57:30.003ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37701 req_id = 4c16a60e-cda7-4428-b47b-e1e4e2753f64 response_code = 200 uri = /network-bootstore-config
22672024-09-11T22:57:31.171ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 637.198097ms file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22682024-09-11T22:57:31.820ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb") failed: Property not found. retry in 1.386998751s file = illumos-utils/src/svc.rs:36 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb zone = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22692024-09-11T22:57:33.238ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb file = sled-agent/src/instance.rs:1466 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22702024-09-11T22:57:33.248ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1475 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22712024-09-11T22:57:33.286ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1499 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22722024-09-11T22:57:33.428ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:744 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22732024-09-11T22:57:33.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3722231 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63015 req_id = ad554282-3c06-4314-a49e-11fbc59170c1 response_code = 200 uri = /vmms/4eb715b5-913d-40f1-9d20-812581b68eeb/state
22742024-09-11T22:57:33.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4525363 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44802 req_id = 83368e46-6337-4f91-8022-e00e330ab041 response_code = 200 uri = /vmms/4eb715b5-913d-40f1-9d20-812581b68eeb/state
22752024-09-11T22:57:33.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3837655 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34553 req_id = 5ed33b85-dc40-420d-b087-a0f4ca1a96cc response_code = 200 uri = /vmms/4eb715b5-913d-40f1-9d20-812581b68eeb/state
22762024-09-11T22:57:33.430ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-09-11T22:57:33.430790599Z } file = sled-agent/src/instance.rs:594 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22772024-09-11T22:57:33.430ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-11T22:57:33.430790599Z } propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22782024-09-11T22:57:33.430ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-11T22:57:33.430790599Z }, migration_in: None, migration_out: None }
22792024-09-11T22:57:33.484ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-09-11T22:57:33.484536489Z } file = sled-agent/src/instance.rs:594 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22802024-09-11T22:57:33.484ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-11T22:57:33.484536489Z } propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22812024-09-11T22:57:33.484ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-11T22:57:33.484536489Z }, migration_in: None, migration_out: None }
22822024-09-11T22:57:39.156ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7163 local_addr = [fd00:1122:3344:101::1]:62558 method = GET remote_addr = [fd00:1122:3344:101::d]:36196 req_id = 6c8bb8fa-1669-40c9-92fe-6a49813bcb26 response_code = 200 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b uri = /35545f26-e1d9-4ad2-8ed3-679cba94411b
22832024-09-11T22:57:43.702ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34600
22842024-09-11T22:57:43.703ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 897 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34600 req_id = 0f8561c8-bac4-4e77-aaa8-57f39b52dcfd response_code = 200 uri = /vmms/4eb715b5-913d-40f1-9d20-812581b68eeb/state
22852024-09-11T22:57:43.703ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-09-11T22:57:43.703708875Z } file = sled-agent/src/instance.rs:594 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22862024-09-11T22:57:43.703ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-11T22:57:43.703708875Z } propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22872024-09-11T22:57:43.703ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-11T22:57:43.703708875Z }, migration_in: None, migration_out: None }
22882024-09-11T22:57:43.997ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-09-11T22:57:43.997635993Z } file = sled-agent/src/instance.rs:594 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22892024-09-11T22:57:43.997ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-09-11T22:57:43.997635993Z } propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22902024-09-11T22:57:43.997ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:630 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22912024-09-11T22:57:43.997ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22922024-09-11T22:57:43.997ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 18069893199645179376 n_samples = 6 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
22932024-09-11T22:57:43.998ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 5877219879681224802 n_samples = 6 sled_id = 35545f26-e1d9-4ad2-8ed3-679cba94411b
22942024-09-11T22:57:44.007ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22952024-09-11T22:57:44.007ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/43105ead-3497-4a46-b5a2-e1fdd8df799c/debug/bundle/zone", "/pool/int/69499c41-13ca-4d82-92e5-3c29ce6f733d/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/7102c444-682c-4072-8e1e-1a1f1e88f906/crypt/debug", "/pool/ext/ac6d979c-58dc-4a4c-94ba-a4cd76c48f8e/crypt/debug", "/pool/ext/f7719a03-4894-4441-833a-ae918464a197/crypt/debug", "/pool/ext/d700c1e0-2586-4b27-b46c-83bfe687db08/crypt/debug", "/pool/ext/8f9c145d-f66d-4533-97ae-72e50686935e/crypt/debug", "/pool/ext/e8902165-fb11-46b2-a335-df247ca55626/crypt/debug", "/pool/ext/7408a7f2-57c1-401a-a865-fdbf6a508b61/crypt/debug", "/pool/ext/cd45e164-586a-42ad-8f77-733daff8a747/crypt/debug", "/pool/ext/2fc72b15-af8d-491c-9d35-a084b85733a3/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb
22962024-09-11T22:57:44.806ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb", bundle_id: 157f34df-48d1-474f-bbc5-b58a4196f3c7 }, time_created: 2024-09-11T22:57:44.007471667Z, version: 0, cause: TerminatedInstance }
22972024-09-11T22:57:44.806ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_4eb715b5-913d-40f1-9d20-812581b68eeb file = sled-agent/src/instance.rs:843 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22982024-09-11T22:57:46.575ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
22992024-09-11T22:57:46.593ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(4856321), kind: System }
23002024-09-11T22:57:46.594ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(4856321), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
23012024-09-11T22:57:46.594ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-11T22:57:46.594320065Z }, migration_in: None, migration_out: None }
23022024-09-11T22:57:46.806ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-11T22:57:46.594320065Z }, migration_in: None, migration_out: None }
23032024-09-11T22:57:46.811ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:780 instance_id = 8375b88a-00e2-48ff-979f-45fa96db29ed propolis_id = 4eb715b5-913d-40f1-9d20-812581b68eeb
23042024-09-11T22:57:47.741ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55443
23052024-09-11T22:57:47.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55443 req_id = f4bd706a-dc57-4b3e-b6ee-47f700020d7b response_code = 200 uri = /vpc-routes
23062024-09-11T22:57:47.751ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 746 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55443 req_id = e6c5044b-8aa7-4082-949c-379e1ba6de54 response_code = 204 uri = /vpc-routes
23072024-09-11T22:57:47.864ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58964
23082024-09-11T22:57:47.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58964 req_id = d73002a0-aa29-42a4-938d-70fc8f1d82e8 response_code = 200 uri = /v2p
23092024-09-11T22:57:48.583ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50485
23102024-09-11T22:57:48.583ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 208 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50485 req_id = 3b7b4fd8-59d6-423a-b828-a059a6efb532 response_code = 200 uri = /v2p
23112024-09-11T22:57:50.365ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54947
23122024-09-11T22:57:50.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54947 req_id = 7baf6395-33ca-4133-b8f6-3d59ff91bed0 response_code = 200 uri = /vpc-routes
23132024-09-11T22:57:50.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54947 req_id = 67b9afc8-91d2-42c6-b912-d8d103ebd17f response_code = 204 uri = /vpc-routes