Max level shown:
1 [ Jun 28 22:08:17 Enabled. ]
2 [ Jun 28 22:08:17 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Jun 28 22:08:17 Method "start" exited with status 0. ]
4 [ Jun 28 22:08:17 Rereading configuration. ]
5 [ Jun 28 22:08:17 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-06-28T22:08:17.451ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-06-28T22:08:17.451ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-06-28T22:08:17.504ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-06-28T22:08:17.504ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-06-28T22:08:17.620ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-06-28T22:08:17.629ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-06-28T22:08:17.637ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-06-28T22:08:18.237ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-06-28T22:08:18.376ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:119
162024-06-28T22:08:18.377ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:131
172024-06-28T22:08:18.377ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
182024-06-28T22:08:18.377ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-06-28T22:08:18.377ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:506
202024-06-28T22:08:18.377ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:625
212024-06-28T22:08:18.380ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
222024-06-28T22:08:18.380ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:674
232024-06-28T22:08:18.380ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
242024-06-28T22:08:18.380ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:733
252024-06-28T22:08:18.380ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_0.vdev
262024-06-28T22:08:18.380ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = None
272024-06-28T22:08:18.380ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
282024-06-28T22:08:18.380ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-06-28T22:08:18.380ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-06-28T22:08:18.392ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:304
312024-06-28T22:08:18.393ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool)
322024-06-28T22:08:18.490ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal }
332024-06-28T22:08:18.692ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
342024-06-28T22:08:18.692ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_1.vdev
352024-06-28T22:08:18.692ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-06-28T22:08:18.692ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-06-28T22:08:18.692ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
382024-06-28T22:08:18.692ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-06-28T22:08:18.692ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-06-28T22:08:18.701ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:304
412024-06-28T22:08:18.701ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool)
422024-06-28T22:08:18.788ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal }
432024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_0.vdev
442024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
452024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
462024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
472024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
482024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_1.vdev
502024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
512024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
522024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
532024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
542024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_2.vdev
562024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
572024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
582024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
592024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_3.vdev
602024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
612024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
622024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
632024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
642024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
652024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
662024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
672024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_4.vdev
682024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
692024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
702024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
712024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
722024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
732024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_5.vdev
742024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
752024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
762024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
772024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
782024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
792024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_6.vdev
802024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
812024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
822024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
832024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
842024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
852024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_7.vdev
862024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
872024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
882024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
892024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
902024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-06-28T22:08:18.989ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_8.vdev
922024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
932024-06-28T22:08:18.989ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
952024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
962024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
972024-06-28T22:08:18.989ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:91
982024-06-28T22:08:18.989ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:93
992024-06-28T22:08:18.989ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
1002024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
1012024-06-28T22:08:18.989ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
1022024-06-28T22:08:18.989ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1032024-06-28T22:08:18.989ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
1042024-06-28T22:08:18.989ZINFOSledAgent (bootstore): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-06-28T22:08:18.989ZINFOSledAgent (bootstore): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-06-28T22:08:18.989ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-06-28T22:08:18.989ZINFOSledAgent (bootstore): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-06-28T22:08:18.989ZINFOSledAgent (bootstore): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-06-28T22:08:18.989ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-06-28T22:08:18.989ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
1112024-06-28T22:08:18.989ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
1122024-06-28T22:08:19.001ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:712
1132024-06-28T22:08:19.001ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1548, tv_nsec: 896919458 } time_to_next_cleanup = 599.99992452s
1142024-06-28T22:08:19.001ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3557
1152024-06-28T22:08:19.001ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1162024-06-28T22:08:19.001ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-06-28T22:08:19.001ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-06-28T22:08:19.004ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-06-28T22:08:19.004ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-06-28T22:08:19.004ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1011
1212024-06-28T22:08:19.004ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-06-28T22:08:19.004ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1232024-06-28T22:08:19.004ZINFOSledAgent (RSS): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-06-28T22:08:19.004ZINFOSledAgent (RSS): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-06-28T22:08:19.004ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1071
1262024-06-28T22:08:19.004ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1272024-06-28T22:08:19.004ZINFOSledAgent (RSS): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-06-28T22:08:19.004ZINFOSledAgent (RSS): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-06-28T22:08:19.004ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1112
1302024-06-28T22:08:19.004ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-06-28T22:08:19.004ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-06-28T22:08:19.004ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1332024-06-28T22:08:19.004ZINFOSledAgent (RSS): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-06-28T22:08:19.005ZINFOSledAgent (RSS): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-06-28T22:08:19.005ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-06-28T22:08:19.005ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1145
1372024-06-28T22:08:19.005ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-06-28T22:08:19.005ZINFOSledAgent (bootstore): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-06-28T22:08:19.005ZINFOSledAgent (bootstore): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-06-28T22:08:19.005ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-06-28T22:08:19.005ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: a9021c63-7f96-431e-9198-9a7654ae0f67, rack_id: 22728d30-b15c-4a22-83e6-f1b1e01718ef, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-06-28T22:08:19.005ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:39830
1432024-06-28T22:08:19.006ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: a9021c63-7f96-431e-9198-9a7654ae0f67, rack_id: 22728d30-b15c-4a22-83e6-f1b1e01718ef, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:360
1442024-06-28T22:08:19.006ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:375
1452024-06-28T22:08:19.006ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1462024-06-28T22:08:19.006ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1472024-06-28T22:08:19.006ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1482024-06-28T22:08:19.006ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1492024-06-28T22:08:19.006ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1502024-06-28T22:08:19.006ZINFOSledAgent (StorageManager): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1512024-06-28T22:08:19.006ZINFOSledAgent (StorageManager): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-06-28T22:08:19.006ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1532024-06-28T22:08:19.006ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1542024-06-28T22:08:19.010ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3789
1552024-06-28T22:08:19.017ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
1562024-06-28T22:08:19.027ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
1572024-06-28T22:08:19.038ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
1582024-06-28T22:08:19.047ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
1592024-06-28T22:08:19.055ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
1602024-06-28T22:08:19.064ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
1612024-06-28T22:08:19.072ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
1622024-06-28T22:08:19.081ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
1632024-06-28T22:08:19.089ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
1642024-06-28T22:08:19.089ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1652024-06-28T22:08:19.089ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1662024-06-28T22:08:19.089ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1672024-06-28T22:08:19.089ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1682024-06-28T22:08:19.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1692024-06-28T22:08:19.100ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:355 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
1702024-06-28T22:08:19.100ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1712024-06-28T22:08:19.100ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:367 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
1722024-06-28T22:08:19.100ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1732024-06-28T22:08:19.103ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 141.634746ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4f01b063-7cc5-4d8c-8008-15ec1b6d7a60", "content-length": "124", "date": "Fri, 28 Jun 2024 22:08:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4f01b063-7cc5-4d8c-8008-15ec1b6d7a60" } file = clients/ddm-admin-client/src/lib.rs:177
1742024-06-28T22:08:19.110ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1752024-06-28T22:08:19.113ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_8cbaafb4-c085-4997-8977-45726fe29b62/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1762024-06-28T22:08:19.128ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1772024-06-28T22:08:19.136ZINFOSledAgent: successfully created encrypted zvol: name="oxi_8cbaafb4-c085-4997-8977-45726fe29b62/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1782024-06-28T22:08:19.136ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_8cbaafb4-c085-4997-8977-45726fe29b62/swap" file = sled-agent/src/swap_device.rs:96
1792024-06-28T22:08:19.141ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:382 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
1802024-06-28T22:08:19.141ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1812024-06-28T22:08:19.141ZINFOSledAgent (BackingFs): Ensuring dataset oxi_8cbaafb4-c085-4997-8977-45726fe29b62/backing/fmd file = sled-agent/src/backing_fs.rs:136
1822024-06-28T22:08:19.212ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1832024-06-28T22:08:19.245ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-06-28T22:08:19.246ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 118.179599ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bd3f297d-0dc3-47d3-8315-70ae0a78642e", "content-length": "124", "date": "Fri, 28 Jun 2024 22:08:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bd3f297d-0dc3-47d3-8315-70ae0a78642e" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-06-28T22:08:19.365ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-06-28T22:08:19.365ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 173.043498ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2bc7f14a-4aba-41f4-9b84-5dcd722b1c86", "content-length": "124", "date": "Fri, 28 Jun 2024 22:08:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2bc7f14a-4aba-41f4-9b84-5dcd722b1c86" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-06-28T22:08:19.431ZINFOSledAgent (BackingFs): Mounting oxi_8cbaafb4-c085-4997-8977-45726fe29b62/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1882024-06-28T22:08:19.443ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1892024-06-28T22:08:19.443ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1902024-06-28T22:08:19.443ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1912024-06-28T22:08:19.443ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1922024-06-28T22:08:19.540ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-06-28T22:08:19.541ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 144.016713ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d9cb5c14-10f9-4555-936d-594d06b2e875", "content-length": "124", "date": "Fri, 28 Jun 2024 22:08:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d9cb5c14-10f9-4555-936d-594d06b2e875" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-06-28T22:08:19.686ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-06-28T22:08:19.687ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 209.558023ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7c82fc1c-d872-4d57-9330-b0e30f4d0294", "content-length": "124", "date": "Fri, 28 Jun 2024 22:08:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7c82fc1c-d872-4d57-9330-b0e30f4d0294" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-06-28T22:08:19.897ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1972024-06-28T22:08:19.897ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 334.494557ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dd34d486-7888-41bd-ab6a-38b1b956de1a", "content-length": "124", "date": "Fri, 28 Jun 2024 22:08:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dd34d486-7888-41bd-ab6a-38b1b956de1a" } file = clients/ddm-admin-client/src/lib.rs:177
1982024-06-28T22:08:20.137ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
1992024-06-28T22:08:20.138ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:41782 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2002024-06-28T22:08:20.138ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2012024-06-28T22:08:20.138ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:41782 file = oximeter/producer/src/lib.rs:282 interval = 30s producer_id = a9021c63-7f96-431e-9198-9a7654ae0f67 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2022024-06-28T22:08:20.144ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 151.22986ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:390 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2032024-06-28T22:08:20.184ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2042024-06-28T22:08:20.184ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2052024-06-28T22:08:20.184ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:895 underlay_address = fd00:1122:3344:101::1
2062024-06-28T22:08:20.184ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2072024-06-28T22:08:20.185ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:12345
2082024-06-28T22:08:20.185ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:430
2092024-06-28T22:08:20.185ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2102024-06-28T22:08:20.185ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2112024-06-28T22:08:20.185ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2122024-06-28T22:08:20.185ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2132024-06-28T22:08:20.185ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:39830
2142024-06-28T22:08:20.185ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2152024-06-28T22:08:20.185ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3557
2162024-06-28T22:08:20.380ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2172024-06-28T22:08:20.382ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2182024-06-28T22:08:20.382ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2192024-06-28T22:08:20.382ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2202024-06-28T22:08:20.382ZINFOSledAgent (RSS): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2212024-06-28T22:08:20.382ZINFOSledAgent (RSS): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2222024-06-28T22:08:20.382ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2232024-06-28T22:08:20.382ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2242024-06-28T22:08:20.383ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 739.254788ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:390 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2252024-06-28T22:08:20.387ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2262024-06-28T22:08:20.414ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61332
2272024-06-28T22:08:20.414ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2282024-06-28T22:08:20.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61332 req_id = 1568a6a9-8eb2-4ac9-8b5d-d6c9840e55d4 response_code = 200 uri = /inventory
2292024-06-28T22:08:20.444ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54942
2302024-06-28T22:08:20.444ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 36 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54942 req_id = e87d8e9e-47f4-474e-abcd-73addcaef966 response_code = 200 uri = /sled-role
2312024-06-28T22:08:20.445ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2322024-06-28T22:08:20.445ZINFOSledAgent (RSS): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2332024-06-28T22:08:20.446ZINFOSledAgent (RSS): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2342024-06-28T22:08:20.446ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:812
2352024-06-28T22:08:20.476ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: 87318ae9-3a3d-4776-b3f0-de10e1f73f8c, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, pool_id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool) }, OmicronPhysicalDiskConfig { id: 8677575f-5429-47de-ac93-f4ca365739cc, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, pool_id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool) }, OmicronPhysicalDiskConfig { id: 7529f46b-55e6-4e53-9c2d-bb36b3bb500d, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, pool_id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool) }, OmicronPhysicalDiskConfig { id: ca77329f-939b-43eb-b5df-6d42a7745830, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, pool_id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool) }, OmicronPhysicalDiskConfig { id: 58be794a-bf87-485d-befa-efafac368661, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, pool_id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool) }, OmicronPhysicalDiskConfig { id: c35c95dc-4b4c-4165-94f6-71d390a64e13, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, pool_id: f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool) }, OmicronPhysicalDiskConfig { id: dfa60738-ac5b-4ad0-9591-68b6c7572ddd, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, pool_id: 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool) }, OmicronPhysicalDiskConfig { id: 0c2d5195-e419-4195-97e1-74edbb022cc0, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, pool_id: c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool) }, OmicronPhysicalDiskConfig { id: 223fbc45-3e93-4324-a362-48995f21ad10, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, pool_id: 04441d44-d54a-4373-b876-f7b81122742f (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:348 sled_address = [fd00:1122:3344:101::1]:12345
2362024-06-28T22:08:20.477ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34818
2372024-06-28T22:08:20.477ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 87318ae9-3a3d-4776-b3f0-de10e1f73f8c, pool_id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 8677575f-5429-47de-ac93-f4ca365739cc, pool_id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 7529f46b-55e6-4e53-9c2d-bb36b3bb500d, pool_id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: ca77329f-939b-43eb-b5df-6d42a7745830, pool_id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 58be794a-bf87-485d-befa-efafac368661, pool_id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: c35c95dc-4b4c-4165-94f6-71d390a64e13, pool_id: f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: dfa60738-ac5b-4ad0-9591-68b6c7572ddd, pool_id: 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 0c2d5195-e419-4195-97e1-74edbb022cc0, pool_id: c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 223fbc45-3e93-4324-a362-48995f21ad10, pool_id: 04441d44-d54a-4373-b876-f7b81122742f (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:381
2382024-06-28T22:08:20.477ZINFOSledAgent (StorageManager): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2392024-06-28T22:08:20.477ZINFOSledAgent (StorageManager): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2402024-06-28T22:08:20.477ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2412024-06-28T22:08:20.477ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2422024-06-28T22:08:20.477ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:379
2432024-06-28T22:08:20.477ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2442024-06-28T22:08:20.477ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2452024-06-28T22:08:20.592ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:304
2462024-06-28T22:08:20.592ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool)
2472024-06-28T22:08:21.260ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.104985273s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:390 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2482024-06-28T22:08:21.260ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2492024-06-28T22:08:21.692ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }
2502024-06-28T22:08:21.891ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:332
2512024-06-28T22:08:21.891ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:334
2522024-06-28T22:08:21.891ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:338
2532024-06-28T22:08:21.891ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:340
2542024-06-28T22:08:21.891ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2552024-06-28T22:08:21.891ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2562024-06-28T22:08:21.891ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2572024-06-28T22:08:22.310ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2582024-06-28T22:08:22.760ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.735012757s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:390 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2592024-06-28T22:08:22.760ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2602024-06-28T22:08:22.760ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2612024-06-28T22:08:22.986ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2622024-06-28T22:08:23.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 65.949032ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2632024-06-28T22:08:23.407ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0/crypt/zone file = sled-storage/src/dataset.rs:395
2642024-06-28T22:08:23.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 110.79338ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2652024-06-28T22:08:24.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 146.803243ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2662024-06-28T22:08:24.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 253.786547ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2672024-06-28T22:08:24.691ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2682024-06-28T22:08:24.691ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2692024-06-28T22:08:24.692ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.19715669s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:390 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2702024-06-28T22:08:25.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 648.021254ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2712024-06-28T22:08:25.694ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2722024-06-28T22:08:26.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.221361547s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2732024-06-28T22:08:26.351ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 file = sled-storage/src/dataset.rs:480
2742024-06-28T22:08:26.539ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2752024-06-28T22:08:26.545ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:477
2762024-06-28T22:08:26.545ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:394
2772024-06-28T22:08:26.545ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:379
2782024-06-28T22:08:26.545ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2792024-06-28T22:08:26.545ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2802024-06-28T22:08:26.756ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:304
2812024-06-28T22:08:26.756ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 120aef62-1850-4042-8634-7cdaae376cbf (zpool)
2822024-06-28T22:08:27.881ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }
2832024-06-28T22:08:28.103ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:332
2842024-06-28T22:08:28.103ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:334
2852024-06-28T22:08:28.103ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:338
2862024-06-28T22:08:28.103ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:340
2872024-06-28T22:08:28.103ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2882024-06-28T22:08:28.103ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2892024-06-28T22:08:28.103ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2902024-06-28T22:08:28.749ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2912024-06-28T22:08:29.196ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2922024-06-28T22:08:29.422ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2932024-06-28T22:08:29.667ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 7.646860474s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:390 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2942024-06-28T22:08:29.881ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_120aef62-1850-4042-8634-7cdaae376cbf/crypt/zone file = sled-storage/src/dataset.rs:395
2952024-06-28T22:08:30.564ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2962024-06-28T22:08:30.566ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2972024-06-28T22:08:30.566ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
2982024-06-28T22:08:30.672ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_120aef62-1850-4042-8634-7cdaae376cbf file = sled-storage/src/dataset.rs:480
2992024-06-28T22:08:30.680ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:477
3002024-06-28T22:08:30.680ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:394
3012024-06-28T22:08:30.680ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:379
3022024-06-28T22:08:30.680ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3032024-06-28T22:08:30.680ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3042024-06-28T22:08:30.690ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:304
3052024-06-28T22:08:30.690ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool)
3062024-06-28T22:08:30.782ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }
3072024-06-28T22:08:30.790ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:332
3082024-06-28T22:08:30.790ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3092024-06-28T22:08:30.790ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3102024-06-28T22:08:30.790ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3112024-06-28T22:08:30.790ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3122024-06-28T22:08:30.790ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3132024-06-28T22:08:30.790ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3142024-06-28T22:08:30.825ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3152024-06-28T22:08:30.834ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_fc1785d4-b751-4b6a-ab11-f04f4f55dd5e/crypt/zone file = sled-storage/src/dataset.rs:395
3162024-06-28T22:08:30.956ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_fc1785d4-b751-4b6a-ab11-f04f4f55dd5e file = sled-storage/src/dataset.rs:480
3172024-06-28T22:08:30.962ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:477
3182024-06-28T22:08:30.962ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:394
3192024-06-28T22:08:30.962ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:379
3202024-06-28T22:08:30.962ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3212024-06-28T22:08:30.962ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3222024-06-28T22:08:30.974ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:304
3232024-06-28T22:08:30.974ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 381e8932-eb2a-4460-814f-428cdbf138af (zpool)
3242024-06-28T22:08:31.069ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External }
3252024-06-28T22:08:31.078ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:332
3262024-06-28T22:08:31.078ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:334
3272024-06-28T22:08:31.078ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:338
3282024-06-28T22:08:31.078ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:340
3292024-06-28T22:08:31.078ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3302024-06-28T22:08:31.079ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3312024-06-28T22:08:31.079ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3322024-06-28T22:08:31.113ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3332024-06-28T22:08:31.121ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_381e8932-eb2a-4460-814f-428cdbf138af/crypt/zone file = sled-storage/src/dataset.rs:395
3342024-06-28T22:08:31.244ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_381e8932-eb2a-4460-814f-428cdbf138af file = sled-storage/src/dataset.rs:480
3352024-06-28T22:08:31.252ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:477
3362024-06-28T22:08:31.252ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:394
3372024-06-28T22:08:31.252ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:379
3382024-06-28T22:08:31.252ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3392024-06-28T22:08:31.252ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3402024-06-28T22:08:31.262ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:304
3412024-06-28T22:08:31.262ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool)
3422024-06-28T22:08:31.353ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External }
3432024-06-28T22:08:31.361ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:332
3442024-06-28T22:08:31.361ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:334
3452024-06-28T22:08:31.361ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:338
3462024-06-28T22:08:31.361ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:340
3472024-06-28T22:08:31.361ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3482024-06-28T22:08:31.362ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3492024-06-28T22:08:31.362ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3502024-06-28T22:08:31.396ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3512024-06-28T22:08:31.403ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f1a7d47d-6c57-42a4-bd15-18672b1e6840/crypt/zone file = sled-storage/src/dataset.rs:395
3522024-06-28T22:08:31.514ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f1a7d47d-6c57-42a4-bd15-18672b1e6840 file = sled-storage/src/dataset.rs:480
3532024-06-28T22:08:31.521ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:477
3542024-06-28T22:08:31.521ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:394
3552024-06-28T22:08:31.521ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:379
3562024-06-28T22:08:31.521ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3572024-06-28T22:08:31.521ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3582024-06-28T22:08:31.530ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:304
3592024-06-28T22:08:31.531ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool)
3602024-06-28T22:08:31.623ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool), kind: External }
3612024-06-28T22:08:31.631ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:332
3622024-06-28T22:08:31.631ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:334
3632024-06-28T22:08:31.631ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:338
3642024-06-28T22:08:31.631ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:340
3652024-06-28T22:08:31.631ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-06-28T22:08:31.631ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3672024-06-28T22:08:31.631ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3682024-06-28T22:08:31.635ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
3692024-06-28T22:08:31.667ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3702024-06-28T22:08:31.675ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f4d2460a-5972-41ce-bacd-ad7cd2f0bd01/crypt/zone file = sled-storage/src/dataset.rs:395
3712024-06-28T22:08:31.797ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 file = sled-storage/src/dataset.rs:480
3722024-06-28T22:08:31.805ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:477
3732024-06-28T22:08:31.805ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:394
3742024-06-28T22:08:31.805ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:379
3752024-06-28T22:08:31.805ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3762024-06-28T22:08:31.805ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3772024-06-28T22:08:31.816ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:304
3782024-06-28T22:08:31.816ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool)
3792024-06-28T22:08:31.910ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool), kind: External }
3802024-06-28T22:08:31.918ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:332
3812024-06-28T22:08:31.918ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:334
3822024-06-28T22:08:31.918ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:338
3832024-06-28T22:08:31.918ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:340
3842024-06-28T22:08:31.918ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3852024-06-28T22:08:31.918ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3862024-06-28T22:08:31.918ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3872024-06-28T22:08:31.928ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2243
3882024-06-28T22:08:31.955ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3892024-06-28T22:08:31.963ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_3090abc7-dfe5-42ac-b133-5d7cdbe91d6e/crypt/zone file = sled-storage/src/dataset.rs:395
3902024-06-28T22:08:32.088ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_3090abc7-dfe5-42ac-b133-5d7cdbe91d6e file = sled-storage/src/dataset.rs:480
3912024-06-28T22:08:32.095ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:477
3922024-06-28T22:08:32.095ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:394
3932024-06-28T22:08:32.095ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:379
3942024-06-28T22:08:32.095ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3952024-06-28T22:08:32.095ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3962024-06-28T22:08:32.104ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:304
3972024-06-28T22:08:32.104ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool)
3982024-06-28T22:08:32.197ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool), kind: External }
3992024-06-28T22:08:32.205ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:332
4002024-06-28T22:08:32.205ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:334
4012024-06-28T22:08:32.205ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:338
4022024-06-28T22:08:32.205ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:340
4032024-06-28T22:08:32.205ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4042024-06-28T22:08:32.205ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4052024-06-28T22:08:32.205ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4062024-06-28T22:08:32.240ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4072024-06-28T22:08:32.247ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c7429e1e-4d01-4919-91c1-3d9c212cba37/crypt/zone file = sled-storage/src/dataset.rs:395
4082024-06-28T22:08:32.368ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c7429e1e-4d01-4919-91c1-3d9c212cba37 file = sled-storage/src/dataset.rs:480
4092024-06-28T22:08:32.376ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:477
4102024-06-28T22:08:32.376ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:394
4112024-06-28T22:08:32.376ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:379
4122024-06-28T22:08:32.376ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
4132024-06-28T22:08:32.376ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4142024-06-28T22:08:32.386ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:304
4152024-06-28T22:08:32.386ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 04441d44-d54a-4373-b876-f7b81122742f (zpool)
4162024-06-28T22:08:32.480ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 04441d44-d54a-4373-b876-f7b81122742f (zpool), kind: External }
4172024-06-28T22:08:32.488ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:332
4182024-06-28T22:08:32.488ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:334
4192024-06-28T22:08:32.488ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:338
4202024-06-28T22:08:32.488ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:340
4212024-06-28T22:08:32.488ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4222024-06-28T22:08:32.488ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4232024-06-28T22:08:32.488ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4242024-06-28T22:08:32.523ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4252024-06-28T22:08:32.531ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_04441d44-d54a-4373-b876-f7b81122742f/crypt/zone file = sled-storage/src/dataset.rs:395
4262024-06-28T22:08:32.567ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
4272024-06-28T22:08:32.637ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
4282024-06-28T22:08:32.652ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_04441d44-d54a-4373-b876-f7b81122742f file = sled-storage/src/dataset.rs:480
4292024-06-28T22:08:32.660ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:477
4302024-06-28T22:08:32.660ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:394
4312024-06-28T22:08:32.660ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4322024-06-28T22:08:32.660ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 8cbaafb4-c085-4997-8977-45726fe29b62 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: aac5d29e-4a80-46cd-84a5-ef3e32cd78c3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: 04441d44-d54a-4373-b876-f7b81122742f (zpool), kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4332024-06-28T22:08:32.660ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
4342024-06-28T22:08:32.660ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
4352024-06-28T22:08:32.660ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4362024-06-28T22:08:32.661ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 12184122 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:34818 req_id = 43f895ea-b409-4a15-a3c1-cdeae12602a9 response_code = 200 uri = /omicron-physical-disks
4372024-06-28T22:08:32.691ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 499354c2-a894-42aa-ba9c-04416f6f33f0, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f604184d-70d9-4043-9c00-ccdd46219e19, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: e3aec3b3-1acd-45de-bcf1-b85068db8902, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
4382024-06-28T22:08:32.691ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58083
4392024-06-28T22:08:32.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4402024-06-28T22:08:32.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4412024-06-28T22:08:32.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4422024-06-28T22:08:32.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }
4432024-06-28T22:08:32.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4442024-06-28T22:08:32.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }
4452024-06-28T22:08:32.837ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4462024-06-28T22:08:32.837ZINFOSledAgent (ServiceManager): No ledger in /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-zones.json file = common/src/ledger.rs:188
4472024-06-28T22:08:32.837ZINFOSledAgent (ServiceManager): No ledger in /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-zones.json file = common/src/ledger.rs:188
4482024-06-28T22:08:32.837ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4492024-06-28T22:08:32.882ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4502024-06-28T22:08:32.882ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4512024-06-28T22:08:32.882ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4522024-06-28T22:08:32.932ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19 file = illumos-utils/src/zone.rs:315
4532024-06-28T22:08:32.936ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902 file = illumos-utils/src/zone.rs:315
4542024-06-28T22:08:32.940ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0 file = illumos-utils/src/zone.rs:315
4552024-06-28T22:08:32.951ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19 file = illumos-utils/src/zone.rs:354
4562024-06-28T22:08:32.955ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902 file = illumos-utils/src/zone.rs:354
4572024-06-28T22:08:32.959ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0 file = illumos-utils/src/zone.rs:354
4582024-06-28T22:08:33.640ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a9021c63-7f96-431e-9198-9a7654ae0f67): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:171 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
4592024-06-28T22:08:34.465ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4602024-06-28T22:08:34.941ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4612024-06-28T22:08:34.941ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4622024-06-28T22:08:34.941ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4632024-06-28T22:08:34.941ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4642024-06-28T22:08:35.427ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4652024-06-28T22:08:35.427ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4662024-06-28T22:08:35.432ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4672024-06-28T22:08:35.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 34.808893ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4682024-06-28T22:08:35.442ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4692024-06-28T22:08:35.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 141.688085ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4702024-06-28T22:08:35.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 62.055444ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4712024-06-28T22:08:35.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 47.793995ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4722024-06-28T22:08:35.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 102.679047ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4732024-06-28T22:08:35.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 84.671587ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4742024-06-28T22:08:35.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 171.452853ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4752024-06-28T22:08:35.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 298.92692ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4762024-06-28T22:08:35.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 198.204656ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4772024-06-28T22:08:35.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 390.886142ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4782024-06-28T22:08:36.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 518.781269ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4792024-06-28T22:08:36.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 327.480236ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4802024-06-28T22:08:36.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 424.934907ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4812024-06-28T22:08:36.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 1.12654581s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4822024-06-28T22:08:36.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 871.404754ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4832024-06-28T22:08:36.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 1.068664292s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4842024-06-28T22:08:37.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 925.738647ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4852024-06-28T22:08:37.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 547.334841ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4862024-06-28T22:08:37.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 894.195108ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4872024-06-28T22:08:37.983ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:9fb2::2 exists in switch zone file = sled-agent/src/services.rs:2260
4882024-06-28T22:08:37.983ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4892024-06-28T22:08:38.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 1.267739588s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
4902024-06-28T22:08:38.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 1.412654599s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
4912024-06-28T22:08:38.526ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fe2b:4a6a file = sled-agent/src/services.rs:2267
4922024-06-28T22:08:38.532ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2328
4932024-06-28T22:08:38.651ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2513
4942024-06-28T22:08:38.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 623.677583ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4952024-06-28T22:08:38.894ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2677
4962024-06-28T22:08:39.082ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2392
4972024-06-28T22:08:39.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19") failed: Property not found. retry in 1.334069716s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
4982024-06-28T22:08:39.376ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2427
4992024-06-28T22:08:39.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 901.292074ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
5002024-06-28T22:08:39.587ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2727
5012024-06-28T22:08:39.769ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2754
5022024-06-28T22:08:39.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0") failed: Property not found. retry in 1.194113008s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
5032024-06-28T22:08:39.983ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2621
5042024-06-28T22:08:40.161ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2421
5052024-06-28T22:08:40.171ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3813 new = [fd00:1122:3344:101::2, ::1] old = [::1]
5062024-06-28T22:08:40.171ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3829
5072024-06-28T22:08:40.171ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
5082024-06-28T22:08:40.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902") failed: Property not found. retry in 605.140517ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
5092024-06-28T22:08:40.725ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3837
5102024-06-28T22:08:40.730ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3893
5112024-06-28T22:08:40.886ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:3955
5122024-06-28T22:08:40.986ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3936 rack_subnet = fd00:1122:3344::
5132024-06-28T22:08:41.009ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:3983
5142024-06-28T22:08:41.062ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_internal_dns_f604184d-70d9-4043-9c00-ccdd46219e19
5152024-06-28T22:08:41.063ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_internal_dns_e3aec3b3-1acd-45de-bcf1-b85068db8902
5162024-06-28T22:08:41.063ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_internal_dns_499354c2-a894-42aa-ba9c-04416f6f33f0
5172024-06-28T22:08:41.063ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5182024-06-28T22:08:41.063ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5192024-06-28T22:08:41.063ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8372169 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:58083 req_id = 530f58c0-4c4a-4a68-8227-508f1c0740a0 response_code = 204 uri = /omicron-zones
5202024-06-28T22:08:41.065ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5212024-06-28T22:08:41.110ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4010
5222024-06-28T22:08:41.229ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:369
5232024-06-28T22:08:41.261ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:417
5242024-06-28T22:08:41.292ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:656
5252024-06-28T22:08:41.292ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:694
5262024-06-28T22:08:41.293ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:697 uptime = 3
5272024-06-28T22:08:41.293ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:439
5282024-06-28T22:08:41.354ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 73.457006ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:569
5292024-06-28T22:08:41.370ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
5302024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5312024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5322024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5332024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5342024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5352024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5362024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5372024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5382024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5392024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5402024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5412024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5422024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5432024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5442024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5452024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5462024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5472024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5482024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5492024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5502024-06-28T22:08:41.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5512024-06-28T22:08:41.371ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5522024-06-28T22:08:41.448ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5532024-06-28T22:08:41.493ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5542024-06-28T22:08:41.539ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:585
5552024-06-28T22:08:41.539ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
5562024-06-28T22:08:41.570ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
5572024-06-28T22:08:41.572ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
5582024-06-28T22:08:41.572ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
5592024-06-28T22:08:41.602ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 499354c2-a894-42aa-ba9c-04416f6f33f0, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f604184d-70d9-4043-9c00-ccdd46219e19, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: e3aec3b3-1acd-45de-bcf1-b85068db8902, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 435dbac3-19be-4fd8-af63-60ea74557b4f, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 23539291-c5e9-4dfc-a999-224bcd6155e8, kind: Service { id: 435dbac3-19be-4fd8-af63-60ea74557b4f }, name: Name("ntp-435dbac3-19be-4fd8-af63-60ea74557b4f"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 233])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
5602024-06-28T22:08:41.603ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48829
5612024-06-28T22:08:41.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5622024-06-28T22:08:41.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5632024-06-28T22:08:41.624ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5642024-06-28T22:08:41.624ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }
5652024-06-28T22:08:41.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5662024-06-28T22:08:41.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }
5672024-06-28T22:08:41.659ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5682024-06-28T22:08:41.659ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-zones.json file = common/src/ledger.rs:177
5692024-06-28T22:08:41.660ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-zones.json file = common/src/ledger.rs:177
5702024-06-28T22:08:41.660ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5712024-06-28T22:08:41.671ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
5722024-06-28T22:08:41.702ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
5732024-06-28T22:08:41.703ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
5742024-06-28T22:08:41.703ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
5752024-06-28T22:08:41.747ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:443 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 169, 233]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" }) }
5762024-06-28T22:08:41.747ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1200 zone_type = ntp
5772024-06-28T22:08:41.747ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5782024-06-28T22:08:41.747ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5792024-06-28T22:08:41.748ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5802024-06-28T22:08:41.750ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5812024-06-28T22:08:41.774ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f file = illumos-utils/src/zone.rs:315
5822024-06-28T22:08:41.792ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f file = illumos-utils/src/zone.rs:354
5832024-06-28T22:08:42.403ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5842024-06-28T22:08:42.403ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5852024-06-28T22:08:42.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 47.008319ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5862024-06-28T22:08:42.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 81.014892ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5872024-06-28T22:08:42.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 107.046512ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5882024-06-28T22:08:42.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 307.891979ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5892024-06-28T22:08:43.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 833.892145ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5902024-06-28T22:08:43.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 1.4298947s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5912024-06-28T22:08:44.653ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42489
5922024-06-28T22:08:44.654ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 100 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42489 req_id = 49d118c0-80b6-49e2-9767-c55871a50a4d response_code = 200 uri = /baseboard
5932024-06-28T22:08:45.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 852.756335ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5942024-06-28T22:08:46.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 559.352538ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5952024-06-28T22:08:46.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f") failed: Property not found. retry in 638.465813ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5962024-06-28T22:08:47.503ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f
5972024-06-28T22:08:47.503ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5982024-06-28T22:08:47.503ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5992024-06-28T22:08:47.504ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 5900897 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48829 req_id = 20fe5f89-fcfe-44da-b207-281182336779 response_code = 204 uri = /omicron-zones
6002024-06-28T22:08:47.505ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:626
6012024-06-28T22:08:47.537ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6022024-06-28T22:08:47.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:32914
6032024-06-28T22:08:47.548ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3543
6042024-06-28T22:08:47.548ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 10483 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:32914 req_id = 3912d030-9af7-4839-bd60-9dabc2a5611b response_code = 500 uri = /timesync
6052024-06-28T22:08:47.549ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6062024-06-28T22:08:47.687ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6072024-06-28T22:08:47.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44887
6082024-06-28T22:08:51.696ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3543
6092024-06-28T22:08:51.696ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 4009167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44887 req_id = 6071820b-e6c2-4882-bbfb-4958960be97b response_code = 500 uri = /timesync
6102024-06-28T22:08:51.700ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6112024-06-28T22:08:51.870ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6122024-06-28T22:08:51.870ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43410
6132024-06-28T22:08:51.878ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3543
6142024-06-28T22:08:51.879ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 8531 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43410 req_id = a34448b7-3fbe-4ace-b44f-6f455ebe0790 response_code = 500 uri = /timesync
6152024-06-28T22:08:51.880ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6162024-06-28T22:08:52.011ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6172024-06-28T22:08:52.011ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48564
6182024-06-28T22:08:56.019ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_435dbac3-19be-4fd8-af63-60ea74557b4f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3543
6192024-06-28T22:08:56.019ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 4007590 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48564 req_id = 15c6eca0-8ff1-43fa-92c4-77a8f8f3aa86 response_code = 500 uri = /timesync
6202024-06-28T22:08:56.023ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6212024-06-28T22:08:56.254ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6222024-06-28T22:08:56.254ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60589
6232024-06-28T22:08:56.263ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8727 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60589 req_id = 857aba78-4ecd-4bef-9d9e-eb643f8f2ff4 response_code = 200 uri = /timesync
6242024-06-28T22:08:56.265ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6252024-06-28T22:08:56.265ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6262024-06-28T22:08:56.410ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6272024-06-28T22:08:56.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61445
6282024-06-28T22:08:56.418ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7972 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61445 req_id = f84367b3-f0fd-4a6f-a30b-9861f3d305ad response_code = 200 uri = /timesync
6292024-06-28T22:08:56.420ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6302024-06-28T22:08:56.420ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6312024-06-28T22:08:56.734ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6322024-06-28T22:08:56.734ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48235
6332024-06-28T22:08:56.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7871 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48235 req_id = a7598a88-2b68-4e58-98a3-e052ed0fc50c response_code = 200 uri = /timesync
6342024-06-28T22:08:56.744ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6352024-06-28T22:08:56.744ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6362024-06-28T22:08:57.062ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6372024-06-28T22:08:57.062ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62289
6382024-06-28T22:08:57.071ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8236 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62289 req_id = 7d80bd83-d1c2-4160-bff9-8b8a3ad23f92 response_code = 200 uri = /timesync
6392024-06-28T22:08:57.074ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6402024-06-28T22:08:57.074ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6412024-06-28T22:08:57.142ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6422024-06-28T22:08:57.142ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56208
6432024-06-28T22:08:57.161ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 18977 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56208 req_id = f754702f-60f8-46ed-be95-4bceb42e478f response_code = 200 uri = /timesync
6442024-06-28T22:08:57.164ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 236.782842ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:423 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
6452024-06-28T22:08:57.165ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 760180647, ip_addr: 45.79.111.167, stratum: 3, ref_time: 1719612536.5486462, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6462024-06-28T22:08:57.165ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1224
6472024-06-28T22:08:57.194ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 499354c2-a894-42aa-ba9c-04416f6f33f0, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f604184d-70d9-4043-9c00-ccdd46219e19, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: e3aec3b3-1acd-45de-bcf1-b85068db8902, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 435dbac3-19be-4fd8-af63-60ea74557b4f, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 23539291-c5e9-4dfc-a999-224bcd6155e8, kind: Service { id: 435dbac3-19be-4fd8-af63-60ea74557b4f }, name: Name("ntp-435dbac3-19be-4fd8-af63-60ea74557b4f"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 233])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 0d99a087-8800-4952-905e-24a2d590a72c, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5fc88cee-9b18-4a01-86bb-51b2cab736c4, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } } } }, OmicronZoneConfig { id: 071e4cf7-62c2-4a8b-845c-aaf52066124b, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } } } }, OmicronZoneConfig { id: f2a4644b-84e2-4183-a96a-df1ad6259bcd, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External } } } }, OmicronZoneConfig { id: 891eb80f-aeb0-4d2d-acea-4310c29853fb, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
6482024-06-28T22:08:57.194ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53097
6492024-06-28T22:08:57.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6502024-06-28T22:08:57.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6512024-06-28T22:08:57.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6522024-06-28T22:08:57.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }
6532024-06-28T22:08:57.232ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6542024-06-28T22:08:57.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }
6552024-06-28T22:08:57.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d99a087-8800-4952-905e-24a2d590a72c, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6562024-06-28T22:08:57.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0d99a087-8800-4952-905e-24a2d590a72c, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6572024-06-28T22:08:57.304ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5fc88cee-9b18-4a01-86bb-51b2cab736c4, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6582024-06-28T22:08:57.304ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5fc88cee-9b18-4a01-86bb-51b2cab736c4, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6592024-06-28T22:08:57.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 071e4cf7-62c2-4a8b-845c-aaf52066124b, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6602024-06-28T22:08:57.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 071e4cf7-62c2-4a8b-845c-aaf52066124b, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6612024-06-28T22:08:57.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f2a4644b-84e2-4183-a96a-df1ad6259bcd, dataset_name: DatasetName { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6622024-06-28T22:08:57.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f2a4644b-84e2-4183-a96a-df1ad6259bcd, dataset_name: DatasetName { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6632024-06-28T22:08:57.461ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 891eb80f-aeb0-4d2d-acea-4310c29853fb, dataset_name: DatasetName { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6642024-06-28T22:08:57.461ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 891eb80f-aeb0-4d2d-acea-4310c29853fb, dataset_name: DatasetName { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6652024-06-28T22:08:57.513ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6662024-06-28T22:08:57.513ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-zones.json file = common/src/ledger.rs:177
6672024-06-28T22:08:57.513ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-zones.json file = common/src/ledger.rs:177
6682024-06-28T22:08:57.513ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6692024-06-28T22:08:57.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6702024-06-28T22:08:57.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6712024-06-28T22:08:57.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6722024-06-28T22:08:57.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6732024-06-28T22:08:57.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6742024-06-28T22:08:57.684ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c file = illumos-utils/src/zone.rs:315
6752024-06-28T22:08:57.689ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb file = illumos-utils/src/zone.rs:315
6762024-06-28T22:08:57.693ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd file = illumos-utils/src/zone.rs:315
6772024-06-28T22:08:57.697ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4 file = illumos-utils/src/zone.rs:315
6782024-06-28T22:08:57.701ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b file = illumos-utils/src/zone.rs:315
6792024-06-28T22:08:57.706ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c file = illumos-utils/src/zone.rs:354
6802024-06-28T22:08:57.710ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb file = illumos-utils/src/zone.rs:354
6812024-06-28T22:08:57.715ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd file = illumos-utils/src/zone.rs:354
6822024-06-28T22:08:57.719ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4 file = illumos-utils/src/zone.rs:354
6832024-06-28T22:08:57.723ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b file = illumos-utils/src/zone.rs:354
6842024-06-28T22:09:00.967ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="891eb80f-aeb0-4d2d-acea-4310c29853fb"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::7]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6852024-06-28T22:09:00.967ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
6862024-06-28T22:09:01.018ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="f2a4644b-84e2-4183-a96a-df1ad6259bcd"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::6]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6872024-06-28T22:09:01.018ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
6882024-06-28T22:09:01.024ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="0d99a087-8800-4952-905e-24a2d590a72c"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::3]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6892024-06-28T22:09:01.024ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
6902024-06-28T22:09:01.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 40.431575ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
6912024-06-28T22:09:01.120ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="5fc88cee-9b18-4a01-86bb-51b2cab736c4"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::4]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6922024-06-28T22:09:01.120ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
6932024-06-28T22:09:01.126ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="071e4cf7-62c2-4a8b-845c-aaf52066124b"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::5]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6942024-06-28T22:09:01.126ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
6952024-06-28T22:09:01.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 73.594754ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
6962024-06-28T22:09:01.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 142.102939ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
6972024-06-28T22:09:01.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 26.327021ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
6982024-06-28T22:09:01.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 110.858712ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
6992024-06-28T22:09:01.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 71.993397ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7002024-06-28T22:09:01.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 223.91497ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7012024-06-28T22:09:01.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 177.810476ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7022024-06-28T22:09:02.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 56.054431ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7032024-06-28T22:09:02.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 131.989011ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7042024-06-28T22:09:02.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 45.69153ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7052024-06-28T22:09:02.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 144.618863ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7062024-06-28T22:09:02.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 420.273275ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7072024-06-28T22:09:02.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 397.597172ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7082024-06-28T22:09:02.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 243.39777ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7092024-06-28T22:09:03.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 124.11542ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7102024-06-28T22:09:03.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 242.991602ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7112024-06-28T22:09:03.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 1.175151091s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7122024-06-28T22:09:03.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 481.556167ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7132024-06-28T22:09:03.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 610.629383ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7142024-06-28T22:09:03.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 149.249879ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7152024-06-28T22:09:03.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 571.173717ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7162024-06-28T22:09:03.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 526.303724ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7172024-06-28T22:09:03.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 904.960667ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7182024-06-28T22:09:04.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 1.320128054s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7192024-06-28T22:09:04.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 824.739129ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7202024-06-28T22:09:04.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 472.704715ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7212024-06-28T22:09:04.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 1.207530237s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7222024-06-28T22:09:04.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 530.076508ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7232024-06-28T22:09:04.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 1.472312667s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7242024-06-28T22:09:05.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 1.071171203s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7252024-06-28T22:09:05.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 1.062063673s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7262024-06-28T22:09:05.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 1.118271893s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7272024-06-28T22:09:05.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 712.390582ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7282024-06-28T22:09:06.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 760.60753ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7292024-06-28T22:09:06.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 699.833463ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7302024-06-28T22:09:06.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 705.399252ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7312024-06-28T22:09:06.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 1.029861925s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7322024-06-28T22:09:06.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 967.877535ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7332024-06-28T22:09:06.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 1.342194629s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7342024-06-28T22:09:07.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 517.158275ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7352024-06-28T22:09:07.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 1.46660853s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7362024-06-28T22:09:07.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 875.265958ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7372024-06-28T22:09:07.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 1.218794189s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7382024-06-28T22:09:07.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 1.203986601s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7392024-06-28T22:09:08.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 1.328500777s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7402024-06-28T22:09:08.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 969.999771ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7412024-06-28T22:09:08.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 857.272181ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7422024-06-28T22:09:08.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 947.291795ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7432024-06-28T22:09:08.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 675.919403ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7442024-06-28T22:09:09.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 1.095037548s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7452024-06-28T22:09:09.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 1.019168039s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7462024-06-28T22:09:09.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 525.739207ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7472024-06-28T22:09:09.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 886.74867ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7482024-06-28T22:09:09.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 795.345965ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7492024-06-28T22:09:10.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 1.242096604s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7502024-06-28T22:09:10.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 1.469795666s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7512024-06-28T22:09:10.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 834.669422ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7522024-06-28T22:09:10.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 932.255409ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7532024-06-28T22:09:10.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 960.808694ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7542024-06-28T22:09:11.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 1.036851431s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7552024-06-28T22:09:11.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4") failed: Property not found. retry in 1.003549195s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7562024-06-28T22:09:11.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd") failed: Property not found. retry in 1.257433366s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7572024-06-28T22:09:11.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 642.574985ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7582024-06-28T22:09:11.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b") failed: Property not found. retry in 1.227243158s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7592024-06-28T22:09:12.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb") failed: Property not found. retry in 617.053961ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7602024-06-28T22:09:12.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c") failed: Property not found. retry in 1.017961334s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7612024-06-28T22:09:12.874ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 512.013429ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:423 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
7622024-06-28T22:09:13.400ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_cockroachdb_0d99a087-8800-4952-905e-24a2d590a72c
7632024-06-28T22:09:13.400ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_cockroachdb_891eb80f-aeb0-4d2d-acea-4310c29853fb
7642024-06-28T22:09:13.400ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_cockroachdb_f2a4644b-84e2-4183-a96a-df1ad6259bcd
7652024-06-28T22:09:13.400ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_cockroachdb_5fc88cee-9b18-4a01-86bb-51b2cab736c4
7662024-06-28T22:09:13.400ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b
7672024-06-28T22:09:13.400ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7682024-06-28T22:09:13.401ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7692024-06-28T22:09:13.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 15782108 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53097 req_id = 02675a1a-464f-491f-b53e-96b01e5c0b5e response_code = 204 uri = /omicron-zones
7702024-06-28T22:09:13.458ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60035
7712024-06-28T22:09:13.487ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 28288 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = af66822c-d410-4f52-8a8a-af10b10c1802 response_code = 500 uri = /cockroachdb
7722024-06-28T22:09:13.487ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "af66822c-d410-4f52-8a8a-af10b10c1802", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "af66822c-d410-4f52-8a8a-af10b10c1802" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 126.822573ms
7732024-06-28T22:09:13.644ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 28675 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = 232020e1-b380-4e7c-b2ee-91954628937a response_code = 500 uri = /cockroachdb
7742024-06-28T22:09:13.644ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "232020e1-b380-4e7c-b2ee-91954628937a", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "232020e1-b380-4e7c-b2ee-91954628937a" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 127.995033ms
7752024-06-28T22:09:13.801ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 27207 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = ea7de911-9a4e-45ad-b3d7-83c636dc2c47 response_code = 500 uri = /cockroachdb
7762024-06-28T22:09:13.801ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ea7de911-9a4e-45ad-b3d7-83c636dc2c47", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ea7de911-9a4e-45ad-b3d7-83c636dc2c47" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 85.947102ms
7772024-06-28T22:09:13.915ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 27163 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = fb1a5ff5-981c-493b-9bbb-935be392143a response_code = 500 uri = /cockroachdb
7782024-06-28T22:09:13.915ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fb1a5ff5-981c-493b-9bbb-935be392143a", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fb1a5ff5-981c-493b-9bbb-935be392143a" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 159.175281ms
7792024-06-28T22:09:14.101ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 26000 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = 6a51e526-cec0-4e2b-8069-218c7d3c3886 response_code = 500 uri = /cockroachdb
7802024-06-28T22:09:14.102ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6a51e526-cec0-4e2b-8069-218c7d3c3886", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6a51e526-cec0-4e2b-8069-218c7d3c3886" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 131.694242ms
7812024-06-28T22:09:14.261ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 26632 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = 2d180d1f-03b6-46d1-b846-6e7cc780cae1 response_code = 500 uri = /cockroachdb
7822024-06-28T22:09:14.262ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2d180d1f-03b6-46d1-b846-6e7cc780cae1", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2d180d1f-03b6-46d1-b846-6e7cc780cae1" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 372.369258ms
7832024-06-28T22:09:14.661ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 25500 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = 6934af1e-19cb-4bda-a7a9-9c303865c197 response_code = 500 uri = /cockroachdb
7842024-06-28T22:09:14.661ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6934af1e-19cb-4bda-a7a9-9c303865c197", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6934af1e-19cb-4bda-a7a9-9c303865c197" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 190.817299ms
7852024-06-28T22:09:14.881ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 27597 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = a553e4df-eb6f-48a0-86e1-a134d0ef44ea response_code = 500 uri = /cockroachdb
7862024-06-28T22:09:14.881ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a553e4df-eb6f-48a0-86e1-a134d0ef44ea", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a553e4df-eb6f-48a0-86e1-a134d0ef44ea" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 321.325607ms
7872024-06-28T22:09:14.921ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61514
7882024-06-28T22:09:14.921ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 49 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61514 req_id = 742e0bb9-4414-493f-a9a0-295f91232299 response_code = 200 uri = /baseboard
7892024-06-28T22:09:15.230ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService9/omicron6 in oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_071e4cf7-62c2-4a8b-845c-aaf52066124b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService9/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 26321 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = db845d32-76c8-44ca-baa4-ba5a4c38657d response_code = 500 uri = /cockroachdb
7902024-06-28T22:09:15.230ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "db845d32-76c8-44ca-baa4-ba5a4c38657d", "content-length": "124", "date": "Fri, 28 Jun 2024 22:09:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "db845d32-76c8-44ca-baa4-ba5a4c38657d" } file = sled-agent/src/rack_setup/service.rs:966 retry_after = 335.430464ms
7912024-06-28T22:09:15.593ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:3377
7922024-06-28T22:09:30.119ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3395
7932024-06-28T22:09:33.696ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3418
7942024-06-28T22:09:33.697ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 18153940 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:60035 req_id = ab5264c4-4f69-4ce6-8773-9e1e9c095b6f response_code = 204 uri = /cockroachdb
7952024-06-28T22:09:33.703ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.222572551s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:423 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
7962024-06-28T22:09:33.735ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 499354c2-a894-42aa-ba9c-04416f6f33f0, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f604184d-70d9-4043-9c00-ccdd46219e19, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: e3aec3b3-1acd-45de-bcf1-b85068db8902, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 435dbac3-19be-4fd8-af63-60ea74557b4f, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 23539291-c5e9-4dfc-a999-224bcd6155e8, kind: Service { id: 435dbac3-19be-4fd8-af63-60ea74557b4f }, name: Name("ntp-435dbac3-19be-4fd8-af63-60ea74557b4f"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 233])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 0d99a087-8800-4952-905e-24a2d590a72c, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5fc88cee-9b18-4a01-86bb-51b2cab736c4, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } } } }, OmicronZoneConfig { id: 071e4cf7-62c2-4a8b-845c-aaf52066124b, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } } } }, OmicronZoneConfig { id: f2a4644b-84e2-4183-a96a-df1ad6259bcd, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External } } } }, OmicronZoneConfig { id: 891eb80f-aeb0-4d2d-acea-4310c29853fb, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External } } } }, OmicronZoneConfig { id: a52585f4-7208-41c3-ac37-3029cbd9b012, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 5ab9e19a-f9b8-42cb-b346-aff1c689c92e, kind: Service { id: a52585f4-7208-41c3-ac37-3029cbd9b012 }, name: Name("external-dns-a52585f4-7208-41c3-ac37-3029cbd9b012"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: effcf430-6675-455f-820f-495829fc7071, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a20b9ed9-42a5-4070-ba7b-11203172977a, kind: Service { id: effcf430-6675-455f-820f-495829fc7071 }, name: Name("external-dns-effcf430-6675-455f-820f-495829fc7071"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 155])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 221b6ce2-61f2-4949-9537-b4af13ae839b, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: f475e35c-414a-4bcc-b0ee-9464f4e4dbaf, kind: Service { id: 221b6ce2-61f2-4949-9537-b4af13ae839b }, name: Name("nexus-221b6ce2-61f2-4949-9537-b4af13ae839b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 119])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 09df608b-9da6-413f-9a4b-1808bcd7a800, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: ba613ac1-2766-48e3-acdb-eb29ca6e137f, kind: Service { id: 09df608b-9da6-413f-9a4b-1808bcd7a800 }, name: Name("nexus-09df608b-9da6-413f-9a4b-1808bcd7a800"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 109])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 35808902-45c0-47d5-8544-cb7fdaca7e8c, kind: Service { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5 }, name: Name("nexus-88974d37-08ca-4371-bfcc-d8e58336bbf5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 69])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: af0a89d1-a224-45ab-9675-7089dedc5fe9, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: b9560b7f-62aa-42db-a117-057b548be234, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } } } }, OmicronZoneConfig { id: a9a229b2-32e9-4ad4-8b94-5a16ef0326c9, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: a92c41e2-55d8-410d-a76b-75046488191b, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 68a9f357-b088-4521-bdfb-3ca7b6381b32, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b24c01d6-5b69-4c4d-87c7-e1ebe641dcad, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 544e2889-6ffb-4271-b2c5-e5814102eec4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c7b01-7629-4266-a541-a595a35b33f0, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External } } } }, OmicronZoneConfig { id: 5b488837-4873-4ac9-bd84-9bee89316c41, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External } } } }, OmicronZoneConfig { id: f1607b64-5448-44df-96f9-c25c1e519af9, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4c85f716-a8e4-4576-8056-ab1ef32e4f68, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7d304db3-3ef6-470f-8340-a6fdffd8f677, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool), kind: External } } } }, OmicronZoneConfig { id: 9c92d46c-84d3-44fe-8227-25882ecb4853, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool), kind: External } } } }, OmicronZoneConfig { id: 885a06f1-aaad-470e-b98a-3c11567fb498, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 04441d44-d54a-4373-b876-f7b81122742f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
7972024-06-28T22:09:33.735ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53048
7982024-06-28T22:09:33.735ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7992024-06-28T22:09:33.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 499354c2-a894-42aa-ba9c-04416f6f33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8002024-06-28T22:09:33.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8012024-06-28T22:09:33.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f604184d-70d9-4043-9c00-ccdd46219e19, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: InternalDns }, responder: ... }
8022024-06-28T22:09:33.778ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8032024-06-28T22:09:33.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: e3aec3b3-1acd-45de-bcf1-b85068db8902, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: InternalDns }, responder: ... }
8042024-06-28T22:09:33.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d99a087-8800-4952-905e-24a2d590a72c, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8052024-06-28T22:09:33.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0d99a087-8800-4952-905e-24a2d590a72c, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8062024-06-28T22:09:33.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5fc88cee-9b18-4a01-86bb-51b2cab736c4, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8072024-06-28T22:09:33.816ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5fc88cee-9b18-4a01-86bb-51b2cab736c4, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8082024-06-28T22:09:33.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 071e4cf7-62c2-4a8b-845c-aaf52066124b, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8092024-06-28T22:09:33.836ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 071e4cf7-62c2-4a8b-845c-aaf52066124b, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8102024-06-28T22:09:33.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f2a4644b-84e2-4183-a96a-df1ad6259bcd, dataset_name: DatasetName { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8112024-06-28T22:09:33.855ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f2a4644b-84e2-4183-a96a-df1ad6259bcd, dataset_name: DatasetName { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8122024-06-28T22:09:33.875ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 891eb80f-aeb0-4d2d-acea-4310c29853fb, dataset_name: DatasetName { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8132024-06-28T22:09:33.875ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 891eb80f-aeb0-4d2d-acea-4310c29853fb, dataset_name: DatasetName { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8142024-06-28T22:09:33.896ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a52585f4-7208-41c3-ac37-3029cbd9b012, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8152024-06-28T22:09:33.896ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a52585f4-7208-41c3-ac37-3029cbd9b012, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8162024-06-28T22:09:33.973ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: effcf430-6675-455f-820f-495829fc7071, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8172024-06-28T22:09:33.973ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: effcf430-6675-455f-820f-495829fc7071, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8182024-06-28T22:09:34.061ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b9560b7f-62aa-42db-a117-057b548be234, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
8192024-06-28T22:09:34.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b9560b7f-62aa-42db-a117-057b548be234, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8202024-06-28T22:09:34.142ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b24c01d6-5b69-4c4d-87c7-e1ebe641dcad, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8212024-06-28T22:09:34.142ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b24c01d6-5b69-4c4d-87c7-e1ebe641dcad, dataset_name: DatasetName { pool_name: ZpoolName { id: 1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0 (zpool), kind: External }, kind: Crucible }, responder: ... }
8222024-06-28T22:09:34.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 544e2889-6ffb-4271-b2c5-e5814102eec4, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8232024-06-28T22:09:34.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 544e2889-6ffb-4271-b2c5-e5814102eec4, dataset_name: DatasetName { pool_name: ZpoolName { id: 120aef62-1850-4042-8634-7cdaae376cbf (zpool), kind: External }, kind: Crucible }, responder: ... }
8242024-06-28T22:09:34.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c7b01-7629-4266-a541-a595a35b33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8252024-06-28T22:09:34.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c7b01-7629-4266-a541-a595a35b33f0, dataset_name: DatasetName { pool_name: ZpoolName { id: fc1785d4-b751-4b6a-ab11-f04f4f55dd5e (zpool), kind: External }, kind: Crucible }, responder: ... }
8262024-06-28T22:09:34.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5b488837-4873-4ac9-bd84-9bee89316c41, dataset_name: DatasetName { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8272024-06-28T22:09:34.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5b488837-4873-4ac9-bd84-9bee89316c41, dataset_name: DatasetName { pool_name: ZpoolName { id: 381e8932-eb2a-4460-814f-428cdbf138af (zpool), kind: External }, kind: Crucible }, responder: ... }
8282024-06-28T22:09:34.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f1607b64-5448-44df-96f9-c25c1e519af9, dataset_name: DatasetName { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8292024-06-28T22:09:34.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f1607b64-5448-44df-96f9-c25c1e519af9, dataset_name: DatasetName { pool_name: ZpoolName { id: f1a7d47d-6c57-42a4-bd15-18672b1e6840 (zpool), kind: External }, kind: Crucible }, responder: ... }
8302024-06-28T22:09:34.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4c85f716-a8e4-4576-8056-ab1ef32e4f68, dataset_name: DatasetName { pool_name: ZpoolName { id: f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8312024-06-28T22:09:34.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4c85f716-a8e4-4576-8056-ab1ef32e4f68, dataset_name: DatasetName { pool_name: ZpoolName { id: f4d2460a-5972-41ce-bacd-ad7cd2f0bd01 (zpool), kind: External }, kind: Crucible }, responder: ... }
8322024-06-28T22:09:34.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7d304db3-3ef6-470f-8340-a6fdffd8f677, dataset_name: DatasetName { pool_name: ZpoolName { id: 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8332024-06-28T22:09:34.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 7d304db3-3ef6-470f-8340-a6fdffd8f677, dataset_name: DatasetName { pool_name: ZpoolName { id: 3090abc7-dfe5-42ac-b133-5d7cdbe91d6e (zpool), kind: External }, kind: Crucible }, responder: ... }
8342024-06-28T22:09:34.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9c92d46c-84d3-44fe-8227-25882ecb4853, dataset_name: DatasetName { pool_name: ZpoolName { id: c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8352024-06-28T22:09:34.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9c92d46c-84d3-44fe-8227-25882ecb4853, dataset_name: DatasetName { pool_name: ZpoolName { id: c7429e1e-4d01-4919-91c1-3d9c212cba37 (zpool), kind: External }, kind: Crucible }, responder: ... }
8362024-06-28T22:09:34.709ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 885a06f1-aaad-470e-b98a-3c11567fb498, dataset_name: DatasetName { pool_name: ZpoolName { id: 04441d44-d54a-4373-b876-f7b81122742f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8372024-06-28T22:09:34.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 885a06f1-aaad-470e-b98a-3c11567fb498, dataset_name: DatasetName { pool_name: ZpoolName { id: 04441d44-d54a-4373-b876-f7b81122742f (zpool), kind: External }, kind: Crucible }, responder: ... }
8382024-06-28T22:09:34.765ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8392024-06-28T22:09:34.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-zones.json file = common/src/ledger.rs:177
8402024-06-28T22:09:34.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-zones.json file = common/src/ledger.rs:177
8412024-06-28T22:09:34.766ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8422024-06-28T22:09:34.886ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
8432024-06-28T22:09:34.887ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8442024-06-28T22:09:34.887ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8452024-06-28T22:09:34.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8462024-06-28T22:09:34.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8472024-06-28T22:09:34.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8482024-06-28T22:09:34.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8492024-06-28T22:09:34.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8502024-06-28T22:09:34.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8512024-06-28T22:09:34.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8522024-06-28T22:09:35.153ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
8532024-06-28T22:09:35.153ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8542024-06-28T22:09:35.153ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8552024-06-28T22:09:35.182ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
8562024-06-28T22:09:35.182ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8572024-06-28T22:09:35.182ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8582024-06-28T22:09:35.182ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8592024-06-28T22:09:35.195ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
8602024-06-28T22:09:35.197ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
8612024-06-28T22:09:35.197ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9 file = illumos-utils/src/zone.rs:315
8622024-06-28T22:09:35.203ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32 file = illumos-utils/src/zone.rs:315
8632024-06-28T22:09:35.208ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68 file = illumos-utils/src/zone.rs:315
8642024-06-28T22:09:35.214ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4 file = illumos-utils/src/zone.rs:315
8652024-06-28T22:09:35.219ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853 file = illumos-utils/src/zone.rs:315
8662024-06-28T22:09:35.224ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498 file = illumos-utils/src/zone.rs:315
8672024-06-28T22:09:35.229ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0 file = illumos-utils/src/zone.rs:315
8682024-06-28T22:09:35.234ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b file = illumos-utils/src/zone.rs:315
8692024-06-28T22:09:35.239ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9 file = illumos-utils/src/zone.rs:315
8702024-06-28T22:09:35.375ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9 file = illumos-utils/src/zone.rs:354
8712024-06-28T22:09:35.381ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32 file = illumos-utils/src/zone.rs:354
8722024-06-28T22:09:35.387ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68 file = illumos-utils/src/zone.rs:354
8732024-06-28T22:09:35.392ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4 file = illumos-utils/src/zone.rs:354
8742024-06-28T22:09:35.399ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853 file = illumos-utils/src/zone.rs:354
8752024-06-28T22:09:35.404ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498 file = illumos-utils/src/zone.rs:354
8762024-06-28T22:09:35.410ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0 file = illumos-utils/src/zone.rs:354
8772024-06-28T22:09:35.415ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b file = illumos-utils/src/zone.rs:354
8782024-06-28T22:09:35.420ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9 file = illumos-utils/src/zone.rs:354
8792024-06-28T22:09:35.426ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234 file = illumos-utils/src/zone.rs:315
8802024-06-28T22:09:35.431ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad file = illumos-utils/src/zone.rs:315
8812024-06-28T22:09:35.438ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41 file = illumos-utils/src/zone.rs:315
8822024-06-28T22:09:35.445ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9 file = illumos-utils/src/zone.rs:315
8832024-06-28T22:09:35.452ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677 file = illumos-utils/src/zone.rs:315
8842024-06-28T22:09:35.460ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234 file = illumos-utils/src/zone.rs:354
8852024-06-28T22:09:35.467ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad file = illumos-utils/src/zone.rs:354
8862024-06-28T22:09:35.474ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41 file = illumos-utils/src/zone.rs:354
8872024-06-28T22:09:35.481ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9 file = illumos-utils/src/zone.rs:354
8882024-06-28T22:09:35.488ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677 file = illumos-utils/src/zone.rs:354
8892024-06-28T22:09:35.553ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
8902024-06-28T22:09:35.630ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
8912024-06-28T22:09:35.782ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
8922024-06-28T22:09:35.786ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
8932024-06-28T22:09:35.786ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
8942024-06-28T22:09:35.921ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:443 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 180, 109]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" }) }
8952024-06-28T22:09:35.921ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1200 zone_type = nexus
8962024-06-28T22:09:35.981ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
8972024-06-28T22:09:36.038ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
8982024-06-28T22:09:36.041ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
8992024-06-28T22:09:36.041ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
9002024-06-28T22:09:36.135ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:443 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 161, 127]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte2" }) }
9012024-06-28T22:09:36.135ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1200 zone_type = external_dns
9022024-06-28T22:09:36.139ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
9032024-06-28T22:09:36.139ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
9042024-06-28T22:09:36.241ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:443 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 253, 69]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte3" }) }
9052024-06-28T22:09:36.242ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1200 zone_type = nexus
9062024-06-28T22:09:36.242ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
9072024-06-28T22:09:36.242ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
9082024-06-28T22:09:36.246ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
9092024-06-28T22:09:36.246ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
9102024-06-28T22:09:36.344ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:443 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 209, 119]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte4" }) }
9112024-06-28T22:09:36.344ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1200 zone_type = nexus
9122024-06-28T22:09:36.345ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
9132024-06-28T22:09:36.345ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
9142024-06-28T22:09:36.347ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
9152024-06-28T22:09:36.347ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
9162024-06-28T22:09:36.447ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:443 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 189, 155]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte5" }) }
9172024-06-28T22:09:36.447ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1200 zone_type = external_dns
9182024-06-28T22:09:36.447ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
9192024-06-28T22:09:36.451ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9202024-06-28T22:09:36.451ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
9212024-06-28T22:09:36.455ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9222024-06-28T22:09:36.539ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
9232024-06-28T22:09:36.539ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
9242024-06-28T22:09:36.539ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012 file = illumos-utils/src/zone.rs:315
9252024-06-28T22:09:36.548ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
9262024-06-28T22:09:36.548ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
9272024-06-28T22:09:36.548ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
9282024-06-28T22:09:36.548ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
9292024-06-28T22:09:36.549ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
9302024-06-28T22:09:36.551ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9312024-06-28T22:09:36.551ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
9322024-06-28T22:09:36.554ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
9332024-06-28T22:09:36.558ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9342024-06-28T22:09:36.558ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9352024-06-28T22:09:36.965ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012 file = illumos-utils/src/zone.rs:354
9362024-06-28T22:09:37.120ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800 file = illumos-utils/src/zone.rs:315
9372024-06-28T22:09:37.231ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5 file = illumos-utils/src/zone.rs:315
9382024-06-28T22:09:37.250ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b file = illumos-utils/src/zone.rs:315
9392024-06-28T22:09:37.442ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800 file = illumos-utils/src/zone.rs:354
9402024-06-28T22:09:37.524ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_effcf430-6675-455f-820f-495829fc7071 file = illumos-utils/src/zone.rs:315
9412024-06-28T22:09:37.536ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5 file = illumos-utils/src/zone.rs:354
9422024-06-28T22:09:37.548ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b file = illumos-utils/src/zone.rs:354
9432024-06-28T22:09:37.567ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_effcf430-6675-455f-820f-495829fc7071 file = illumos-utils/src/zone.rs:354
9442024-06-28T22:09:38.987ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9452024-06-28T22:09:38.988ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9462024-06-28T22:09:38.988ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
9472024-06-28T22:09:38.995ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
9482024-06-28T22:09:39.075ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_120aef62-1850-4042-8634-7cdaae376cbf/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="1624fe10-4b40-4d42-b445-9c82186d66a1"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9492024-06-28T22:09:39.076ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
9502024-06-28T22:09:39.095ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_c7429e1e-4d01-4919-91c1-3d9c212cba37/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="ec578f70-3110-4aae-a64b-c06154628852"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9512024-06-28T22:09:39.099ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_04441d44-d54a-4373-b876-f7b81122742f/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="f083cdb6-891c-490d-960c-ec69caaadf51"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9522024-06-28T22:09:39.103ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_fc1785d4-b751-4b6a-ab11-f04f4f55dd5e/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="53c0b3eb-534a-460b-b704-7a925e51c4a3"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9532024-06-28T22:09:39.107ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_1ee6ce71-47f2-4d3e-acc7-b1cedfc662c0/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="90e0663d-43ca-4438-a00d-da52cf6de81b"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9542024-06-28T22:09:39.108ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_381e8932-eb2a-4460-814f-428cdbf138af/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="81f84d61-b5d6-4af2-87da-104bfce0405e"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9552024-06-28T22:09:39.108ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_3090abc7-dfe5-42ac-b133-5d7cdbe91d6e/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="bc59b127-25f3-4417-8093-94217adc4813"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9562024-06-28T22:09:39.108ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_f4d2460a-5972-41ce-bacd-ad7cd2f0bd01/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="4c063020-b2b6-46df-949d-cfda6c4c683a"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9572024-06-28T22:09:39.108ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
9582024-06-28T22:09:39.116ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
9592024-06-28T22:09:39.124ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
9602024-06-28T22:09:39.131ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
9612024-06-28T22:09:39.137ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
9622024-06-28T22:09:39.144ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
9632024-06-28T22:09:39.152ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
9642024-06-28T22:09:39.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 49.14246ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
9652024-06-28T22:09:39.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 67.095771ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
9662024-06-28T22:09:39.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 30.437635ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
9672024-06-28T22:09:39.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 139.230052ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
9682024-06-28T22:09:40.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 131.39125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
9692024-06-28T22:09:40.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 148.839426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
9702024-06-28T22:09:40.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 34.236238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
9712024-06-28T22:09:40.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 52.071255ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
9722024-06-28T22:09:40.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 63.355114ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
9732024-06-28T22:09:40.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 161.36977ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
9742024-06-28T22:09:40.808ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 60.003828ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
9752024-06-28T22:09:41.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 31.400769ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
9762024-06-28T22:09:42.479ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9772024-06-28T22:09:42.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 31.771554ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
9782024-06-28T22:09:43.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 153.439523ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
9792024-06-28T22:09:43.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 69.352325ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
9802024-06-28T22:09:43.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 164.833015ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
9812024-06-28T22:09:43.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 110.689004ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
9822024-06-28T22:09:43.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 79.047472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
9832024-06-28T22:09:43.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 133.369656ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
9842024-06-28T22:09:43.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 212.830527ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
9852024-06-28T22:09:43.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 92.536312ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
9862024-06-28T22:09:44.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 111.1389ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
9872024-06-28T22:09:44.024ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
9882024-06-28T22:09:44.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 108.798528ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
9892024-06-28T22:09:44.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 558.996952ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
9902024-06-28T22:09:44.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 87.032438ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
9912024-06-28T22:09:44.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 508.439266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
9922024-06-28T22:09:44.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 131.628129ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
9932024-06-28T22:09:44.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 172.971887ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
9942024-06-28T22:09:44.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 123.738619ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
9952024-06-28T22:09:44.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 575.356964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
9962024-06-28T22:09:44.200ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="af0a89d1-a224-45ab-9675-7089dedc5fe9"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9972024-06-28T22:09:44.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 264.890866ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
9982024-06-28T22:09:44.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 110.755329ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
9992024-06-28T22:09:44.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 115.945366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
10002024-06-28T22:09:44.239ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_f1a7d47d-6c57-42a4-bd15-18672b1e6840/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="e7508e08-99f9-4415-af90-9a1c8e44e379"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10012024-06-28T22:09:44.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 152.475645ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
10022024-06-28T22:09:44.251ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10032024-06-28T22:09:44.257ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10042024-06-28T22:09:44.308ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 560.684163ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
10052024-06-28T22:09:44.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 508.968267ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
10062024-06-28T22:09:44.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 207.31834ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
10072024-06-28T22:09:44.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 275.536051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
10082024-06-28T22:09:44.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 55.585768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10092024-06-28T22:09:44.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 251.088745ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
10102024-06-28T22:09:44.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 555.160104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
10112024-06-28T22:09:44.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 106.21556ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10122024-06-28T22:09:44.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 25.057329ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10132024-06-28T22:09:44.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 424.500756ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
10142024-06-28T22:09:44.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 37.83238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10152024-06-28T22:09:44.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 71.29315ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10162024-06-28T22:09:44.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 67.900722ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10172024-06-28T22:09:44.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 436.494895ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
10182024-06-28T22:09:44.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 261.119491ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10192024-06-28T22:09:44.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 273.490252ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10202024-06-28T22:09:44.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 855.831855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
10212024-06-28T22:09:44.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 150.922048ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10222024-06-28T22:09:44.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 503.55652ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
10232024-06-28T22:09:44.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.148854487s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
10242024-06-28T22:09:44.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.164363789s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
10252024-06-28T22:09:44.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.332348009s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
10262024-06-28T22:09:44.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.162051005s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
10272024-06-28T22:09:44.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 578.45089ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10282024-06-28T22:09:44.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 981.160293ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
10292024-06-28T22:09:44.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 714.073674ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
10302024-06-28T22:09:44.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 313.040339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10312024-06-28T22:09:45.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 516.759241ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
10322024-06-28T22:09:45.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 585.676464ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10332024-06-28T22:09:45.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 799.607427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
10342024-06-28T22:09:45.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.438610304s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
10352024-06-28T22:09:45.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.132979678s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10362024-06-28T22:09:45.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 657.552653ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10372024-06-28T22:09:45.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 1.00142623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
10382024-06-28T22:09:45.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.389632378s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
10392024-06-28T22:09:45.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.097547888s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10402024-06-28T22:09:45.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 505.119692ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
10412024-06-28T22:09:45.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.448846164s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
10422024-06-28T22:09:45.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 773.226484ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
10432024-06-28T22:09:45.949ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46700
10442024-06-28T22:09:45.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.274427656s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
10452024-06-28T22:09:45.962ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 11202 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46700 req_id = 04747f48-5258-468e-b90f-c96a35f4ff92 response_code = 200 uri = /baseboard
10462024-06-28T22:09:45.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 899.308944ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
10472024-06-28T22:09:46.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.351382435s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
10482024-06-28T22:09:46.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 1.490945197s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10492024-06-28T22:09:46.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.05782328s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
10502024-06-28T22:09:46.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.056472271s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
10512024-06-28T22:09:46.298ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_effcf430-6675-455f-820f-495829fc7071: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte5"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10522024-06-28T22:09:46.299ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
10532024-06-28T22:09:46.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 672.083168ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10542024-06-28T22:09:46.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 1.255464008s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
10552024-06-28T22:09:46.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 49.803192ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
10562024-06-28T22:09:46.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.212804399s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
10572024-06-28T22:09:46.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 92.516912ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
10582024-06-28T22:09:46.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 588.705064ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
10592024-06-28T22:09:46.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 964.344239ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10602024-06-28T22:09:46.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 240.864399ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
10612024-06-28T22:09:46.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 653.926595ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
10622024-06-28T22:09:46.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.137837263s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
10632024-06-28T22:09:47.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 319.596517ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
10642024-06-28T22:09:47.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 914.210863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10652024-06-28T22:09:47.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 733.496497ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
10662024-06-28T22:09:47.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.344189068s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
10672024-06-28T22:09:47.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.475501363s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
10682024-06-28T22:09:47.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.199823632s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
10692024-06-28T22:09:47.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 610.285941ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
10702024-06-28T22:09:47.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 968.99905ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
10712024-06-28T22:09:47.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.127650071s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
10722024-06-28T22:09:47.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 775.138105ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
10732024-06-28T22:09:47.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 724.362679ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10742024-06-28T22:09:47.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 630.291407ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10752024-06-28T22:09:47.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 862.135181ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
10762024-06-28T22:09:47.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 933.715834ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
10772024-06-28T22:09:47.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 548.475493ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
10782024-06-28T22:09:48.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.287843724s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
10792024-06-28T22:09:48.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 785.147285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
10802024-06-28T22:09:48.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.15178898s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
10812024-06-28T22:09:48.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 581.308278ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
10822024-06-28T22:09:48.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.080644413s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
10832024-06-28T22:09:48.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 1.304272256s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
10842024-06-28T22:09:48.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.433454006s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
10852024-06-28T22:09:48.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 870.418026ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
10862024-06-28T22:09:48.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 593.845657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
10872024-06-28T22:09:48.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 751.477994ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
10882024-06-28T22:09:48.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.261959761s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
10892024-06-28T22:09:48.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 771.903414ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
10902024-06-28T22:09:48.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.022071193s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
10912024-06-28T22:09:48.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 942.856059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
10922024-06-28T22:09:48.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 634.111581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
10932024-06-28T22:09:48.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.351852424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
10942024-06-28T22:09:48.995ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10952024-06-28T22:09:48.996ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
10962024-06-28T22:09:49.001ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte4"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10972024-06-28T22:09:49.001ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
10982024-06-28T22:09:49.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 38.729288ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
10992024-06-28T22:09:49.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 52.342854ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11002024-06-28T22:09:49.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 123.245052ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11012024-06-28T22:09:49.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.36921843s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
11022024-06-28T22:09:49.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 87.623901ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11032024-06-28T22:09:49.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.405973642s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
11042024-06-28T22:09:49.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.028500495s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
11052024-06-28T22:09:49.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 155.997329ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11062024-06-28T22:09:49.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 268.100627ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11072024-06-28T22:09:49.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.150501627s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
11082024-06-28T22:09:49.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 683.113302ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
11092024-06-28T22:09:49.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 714.443144ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
11102024-06-28T22:09:49.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 445.092166ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11112024-06-28T22:09:49.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 538.884491ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
11122024-06-28T22:09:49.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.165519467s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
11132024-06-28T22:09:49.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 551.418016ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11142024-06-28T22:09:49.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 527.299472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
11152024-06-28T22:09:49.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 519.944128ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
11162024-06-28T22:09:49.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 797.757438ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
11172024-06-28T22:09:49.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.190712841s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
11182024-06-28T22:09:49.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 775.800099ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
11192024-06-28T22:09:49.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.196452258s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11202024-06-28T22:09:51.046ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.075755723s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:423 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
11212024-06-28T22:09:51.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 771.954968ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
11222024-06-28T22:09:51.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 969.848454ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
11232024-06-28T22:09:51.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 803.848776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
11242024-06-28T22:09:51.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.105790574s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
11252024-06-28T22:09:51.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 549.614477ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
11262024-06-28T22:09:51.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.09237205s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
11272024-06-28T22:09:51.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 688.39125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
11282024-06-28T22:09:51.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 859.646965ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
11292024-06-28T22:09:51.113ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 808.021141ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
11302024-06-28T22:09:51.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 843.863171ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
11312024-06-28T22:09:51.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 492.068474ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11322024-06-28T22:09:51.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 763.10735ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
11332024-06-28T22:09:51.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 929.729674ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
11342024-06-28T22:09:51.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 653.863896ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
11352024-06-28T22:09:51.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.174922654s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11362024-06-28T22:09:51.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 651.755778ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
11372024-06-28T22:09:51.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.324325237s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11382024-06-28T22:09:51.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.02118725s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
11392024-06-28T22:09:51.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 800.672519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
11402024-06-28T22:09:51.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.203969187s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
11412024-06-28T22:09:51.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 760.671718ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
11422024-06-28T22:09:51.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 928.645354ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
11432024-06-28T22:09:51.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 702.312413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
11442024-06-28T22:09:51.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 969.879889ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
11452024-06-28T22:09:51.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.393727904s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
11462024-06-28T22:09:51.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 810.007461ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
11472024-06-28T22:09:51.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.430590823s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
11482024-06-28T22:09:52.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 880.477973ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
11492024-06-28T22:09:52.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.363268147s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
11502024-06-28T22:09:52.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 707.989238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
11512024-06-28T22:09:52.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.395526165s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
11522024-06-28T22:09:52.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 683.172204ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11532024-06-28T22:09:52.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 1.060999247s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
11542024-06-28T22:09:52.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 629.047871ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
11552024-06-28T22:09:52.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 624.311643ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
11562024-06-28T22:09:52.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.11344989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
11572024-06-28T22:09:52.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 1.279617707s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
11582024-06-28T22:09:52.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.051222432s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
11592024-06-28T22:09:52.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 566.412517ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
11602024-06-28T22:09:52.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 740.37949ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
11612024-06-28T22:09:52.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.424540957s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11622024-06-28T22:09:52.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 798.021299ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
11632024-06-28T22:09:53.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.271743591s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
11642024-06-28T22:09:53.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.327499301s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11652024-06-28T22:09:53.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.416665002s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
11662024-06-28T22:09:53.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.388273319s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
11672024-06-28T22:09:53.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 513.355304ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
11682024-06-28T22:09:53.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 814.02164ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
11692024-06-28T22:09:53.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.47022847s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
11702024-06-28T22:09:53.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.123033981s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
11712024-06-28T22:09:53.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 892.931512ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
11722024-06-28T22:09:53.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.209726012s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
11732024-06-28T22:09:53.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 624.115355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
11742024-06-28T22:09:53.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 877.664623ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
11752024-06-28T22:09:53.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 812.453957ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
11762024-06-28T22:09:53.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 979.607938ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
11772024-06-28T22:09:53.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.084587745s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
11782024-06-28T22:09:54.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 519.936686ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
11792024-06-28T22:09:54.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.359509588s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
11802024-06-28T22:09:54.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 692.81498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
11812024-06-28T22:09:54.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 626.637155ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
11822024-06-28T22:09:54.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.295981339s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
11832024-06-28T22:09:54.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.332293567s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
11842024-06-28T22:09:54.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 1.400398743s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
11852024-06-28T22:09:54.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 876.584436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
11862024-06-28T22:09:54.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 1.049957173s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
11872024-06-28T22:09:54.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.018577128s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
11882024-06-28T22:09:54.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 874.086077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
11892024-06-28T22:09:54.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 535.221237ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
11902024-06-28T22:09:54.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 503.693327ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
11912024-06-28T22:09:54.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 550.658138ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
11922024-06-28T22:09:54.848ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte2"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11932024-06-28T22:09:54.848ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
11942024-06-28T22:09:54.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 591.07949ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
11952024-06-28T22:09:54.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 742.679207ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
11962024-06-28T22:09:54.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 644.194961ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
11972024-06-28T22:09:55.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 616.11659ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
11982024-06-28T22:09:55.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 37.428789ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
11992024-06-28T22:09:55.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 543.629878ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
12002024-06-28T22:09:55.078ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 95.596456ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12012024-06-28T22:09:55.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 153.270748ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12022024-06-28T22:09:55.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 902.428578ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
12032024-06-28T22:09:55.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.380404606s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
12042024-06-28T22:09:55.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 376.463441ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12052024-06-28T22:09:55.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 705.851791ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
12062024-06-28T22:09:55.477ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte3"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12072024-06-28T22:09:55.478ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12082024-06-28T22:09:55.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.163845879s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
12092024-06-28T22:09:55.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.233164754s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
12102024-06-28T22:09:55.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.491330623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
12112024-06-28T22:09:55.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.192746845s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
12122024-06-28T22:09:55.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.309329646s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
12132024-06-28T22:09:55.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 72.990068ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12142024-06-28T22:09:55.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.085119263s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
12152024-06-28T22:09:55.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 826.667928ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
12162024-06-28T22:09:55.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.156243633s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
12172024-06-28T22:09:55.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 1.264658311s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
12182024-06-28T22:09:55.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 584.743228ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
12192024-06-28T22:09:55.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 701.048179ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
12202024-06-28T22:09:55.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 89.90893ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12212024-06-28T22:09:55.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 663.484954ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12222024-06-28T22:09:55.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.137113085s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
12232024-06-28T22:09:55.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 218.885303ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12242024-06-28T22:09:55.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 813.639398ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
12252024-06-28T22:09:56.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 290.038657ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12262024-06-28T22:09:56.113ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 891.83765ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
12272024-06-28T22:09:56.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.388198592s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
12282024-06-28T22:09:56.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.499532756s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
12292024-06-28T22:09:56.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 778.332643ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12302024-06-28T22:09:56.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 1.297920098s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12312024-06-28T22:09:56.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.397638223s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
12322024-06-28T22:09:56.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 788.506518ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
12332024-06-28T22:09:56.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 790.627259ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
12342024-06-28T22:09:56.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.325664719s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
12352024-06-28T22:09:56.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.271314299s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
12362024-06-28T22:09:56.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.11469827s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
12372024-06-28T22:09:56.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.00756392s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
12382024-06-28T22:09:56.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 1.434898209s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
12392024-06-28T22:09:56.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 853.548942ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
12402024-06-28T22:09:56.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 731.562242ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
12412024-06-28T22:09:56.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.087686366s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
12422024-06-28T22:09:56.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 503.939478ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
12432024-06-28T22:09:57.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.242905958s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
12442024-06-28T22:09:57.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.157531372s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
12452024-06-28T22:09:57.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 846.34211ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12462024-06-28T22:09:57.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 916.961954ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
12472024-06-28T22:09:57.310ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12482024-06-28T22:09:57.311ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
12492024-06-28T22:09:57.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.110840506s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
12502024-06-28T22:09:57.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 583.962139ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
12512024-06-28T22:09:57.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 34.889619ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
12522024-06-28T22:09:57.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.377765886s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
12532024-06-28T22:09:57.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 74.34436ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
12542024-06-28T22:09:57.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 203.855274ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
12552024-06-28T22:09:57.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.169185102s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
12562024-06-28T22:09:57.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.134665218s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
12572024-06-28T22:09:57.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 1.190226685s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12582024-06-28T22:09:57.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.148740155s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
12592024-06-28T22:09:57.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.135150676s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
12602024-06-28T22:09:57.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 240.635245ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
12612024-06-28T22:09:57.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.260208896s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
12622024-06-28T22:09:57.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.296551915s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
12632024-06-28T22:09:58.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.222967409s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
12642024-06-28T22:09:58.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 595.736548ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12652024-06-28T22:09:58.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.103549484s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
12662024-06-28T22:09:58.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.029932822s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
12672024-06-28T22:09:58.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 624.698117ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
12682024-06-28T22:09:58.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 778.077192ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
12692024-06-28T22:09:58.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 925.519751ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
12702024-06-28T22:09:58.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.132224935s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
12712024-06-28T22:09:58.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 1.490259445s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
12722024-06-28T22:09:58.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 933.369814ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
12732024-06-28T22:09:58.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 613.35033ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
12742024-06-28T22:09:58.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 1.006404871s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12752024-06-28T22:09:58.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 950.877009ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
12762024-06-28T22:09:58.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.347867626s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
12772024-06-28T22:09:58.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 776.550031ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
12782024-06-28T22:09:58.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 1.041367531s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
12792024-06-28T22:09:58.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.02976604s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
12802024-06-28T22:09:58.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 600.697529ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12812024-06-28T22:09:58.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.008589989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
12822024-06-28T22:09:59.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.425922386s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
12832024-06-28T22:09:59.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 790.792619ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
12842024-06-28T22:09:59.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 875.187462ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
12852024-06-28T22:09:59.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.131601829s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
12862024-06-28T22:09:59.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.336684004s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
12872024-06-28T22:09:59.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.20372284s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
12882024-06-28T22:09:59.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.374030905s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
12892024-06-28T22:09:59.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 637.76123ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
12902024-06-28T22:09:59.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.463672636s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
12912024-06-28T22:09:59.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 980.470205ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
12922024-06-28T22:09:59.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 1.020584686s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
12932024-06-28T22:09:59.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 660.832236ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
12942024-06-28T22:09:59.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 707.267595ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
12952024-06-28T22:09:59.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 539.917999ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
12962024-06-28T22:09:59.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 795.742498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
12972024-06-28T22:09:59.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 661.17022ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
12982024-06-28T22:09:59.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 596.215801ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
12992024-06-28T22:09:59.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 825.348841ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
13002024-06-28T22:09:59.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.243293659s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
13012024-06-28T22:10:00.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.376801391s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
13022024-06-28T22:10:00.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 670.839262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
13032024-06-28T22:10:00.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.434126549s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
13042024-06-28T22:10:00.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 1.155222673s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
13052024-06-28T22:10:00.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 880.227471ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
13062024-06-28T22:10:00.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 510.621188ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
13072024-06-28T22:10:00.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.424713431s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
13082024-06-28T22:10:00.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 800.496147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
13092024-06-28T22:10:00.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 741.473904ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
13102024-06-28T22:10:00.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 680.112539ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
13112024-06-28T22:10:00.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 643.62773ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
13122024-06-28T22:10:00.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.390790782s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
13132024-06-28T22:10:00.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.346623782s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
13142024-06-28T22:10:00.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 850.461466ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
13152024-06-28T22:10:00.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 1.018124463s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
13162024-06-28T22:10:00.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.37389553s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
13172024-06-28T22:10:00.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.46725747s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
13182024-06-28T22:10:00.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.113836156s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
13192024-06-28T22:10:00.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 1.124597856s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
13202024-06-28T22:10:00.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 1.05364979s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
13212024-06-28T22:10:01.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 975.131521ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
13222024-06-28T22:10:01.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.307336416s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
13232024-06-28T22:10:01.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 611.205707ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
13242024-06-28T22:10:01.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.346004047s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
13252024-06-28T22:10:01.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 644.533476ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
13262024-06-28T22:10:01.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 727.776352ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
13272024-06-28T22:10:01.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 875.176849ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
13282024-06-28T22:10:01.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 921.795375ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
13292024-06-28T22:10:01.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 751.059699ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
13302024-06-28T22:10:01.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 1.133200274s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
13312024-06-28T22:10:01.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.316988947s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
13322024-06-28T22:10:01.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.043695952s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
13332024-06-28T22:10:01.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.27262789s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
13342024-06-28T22:10:01.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.408375463s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
13352024-06-28T22:10:01.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.361150533s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
13362024-06-28T22:10:01.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 1.383980923s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
13372024-06-28T22:10:01.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.11004985s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
13382024-06-28T22:10:01.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 861.26392ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
13392024-06-28T22:10:01.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 984.610913ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
13402024-06-28T22:10:01.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 547.597039ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
13412024-06-28T22:10:01.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 623.190629ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
13422024-06-28T22:10:02.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.089186581s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
13432024-06-28T22:10:02.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.077438751s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
13442024-06-28T22:10:02.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 801.301861ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
13452024-06-28T22:10:02.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.411669351s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
13462024-06-28T22:10:02.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 518.787294ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
13472024-06-28T22:10:02.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.427369052s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
13482024-06-28T22:10:02.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 596.563722ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
13492024-06-28T22:10:02.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.310012881s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
13502024-06-28T22:10:02.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.475109231s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
13512024-06-28T22:10:02.770ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 1.296572935s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
13522024-06-28T22:10:02.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 848.796134ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
13532024-06-28T22:10:02.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 999.765062ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
13542024-06-28T22:10:02.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 534.871791ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
13552024-06-28T22:10:02.956ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.053608693s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
13562024-06-28T22:10:02.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 933.205171ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
13572024-06-28T22:10:03.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 533.499616ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
13582024-06-28T22:10:03.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.247290792s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
13592024-06-28T22:10:03.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 1.060606515s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
13602024-06-28T22:10:03.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.459060285s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
13612024-06-28T22:10:03.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 1.374285139s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
13622024-06-28T22:10:03.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 802.797663ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
13632024-06-28T22:10:03.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 853.770097ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
13642024-06-28T22:10:03.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.2075901s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
13652024-06-28T22:10:03.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 1.007532165s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
13662024-06-28T22:10:03.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 1.468994746s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
13672024-06-28T22:10:03.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 831.206132ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
13682024-06-28T22:10:03.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 838.186816ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
13692024-06-28T22:10:03.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 1.383655031s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
13702024-06-28T22:10:03.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.468599754s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
13712024-06-28T22:10:03.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 1.137465714s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
13722024-06-28T22:10:03.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 1.11624994s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
13732024-06-28T22:10:03.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.038547043s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
13742024-06-28T22:10:04.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800") failed: Property not found. retry in 1.439226576s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
13752024-06-28T22:10:04.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 580.83415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
13762024-06-28T22:10:04.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012") failed: Property not found. retry in 1.2766512s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
13772024-06-28T22:10:04.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 883.222915ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
13782024-06-28T22:10:04.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b") failed: Property not found. retry in 1.026626366s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
13792024-06-28T22:10:04.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 625.974611ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
13802024-06-28T22:10:04.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32") failed: Property not found. retry in 1.435288659s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
13812024-06-28T22:10:04.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 573.621172ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
13822024-06-28T22:10:04.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68") failed: Property not found. retry in 1.421935716s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
13832024-06-28T22:10:04.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41") failed: Property not found. retry in 1.282687037s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
13842024-06-28T22:10:04.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234") failed: Property not found. retry in 1.385879136s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
13852024-06-28T22:10:04.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677") failed: Property not found. retry in 926.444023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
13862024-06-28T22:10:04.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad") failed: Property not found. retry in 1.446578378s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
13872024-06-28T22:10:04.653ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 522.454147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
13882024-06-28T22:10:04.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_effcf430-6675-455f-820f-495829fc7071") failed: Property not found. retry in 666.832719ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
13892024-06-28T22:10:04.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9") failed: Property not found. retry in 688.644498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
13902024-06-28T22:10:04.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5") failed: Property not found. retry in 1.49492944s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
13912024-06-28T22:10:05.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9") failed: Property not found. retry in 1.322717867s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
13922024-06-28T22:10:05.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4") failed: Property not found. retry in 1.335246354s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
13932024-06-28T22:10:05.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0") failed: Property not found. retry in 500.808681ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
13942024-06-28T22:10:05.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b") failed: Property not found. retry in 818.314757ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
13952024-06-28T22:10:05.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9") failed: Property not found. retry in 809.293132ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
13962024-06-28T22:10:05.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853") failed: Property not found. retry in 1.165548549s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
13972024-06-28T22:10:05.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498") failed: Property not found. retry in 1.399277021s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
13982024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_f1607b64-5448-44df-96f9-c25c1e519af9
13992024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_pantry_68a9f357-b088-4521-bdfb-3ca7b6381b32
14002024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_external_dns_a52585f4-7208-41c3-ac37-3029cbd9b012
14012024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_4c85f716-a8e4-4576-8056-ab1ef32e4f68
14022024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_544e2889-6ffb-4271-b2c5-e5814102eec4
14032024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_9c92d46c-84d3-44fe-8227-25882ecb4853
14042024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_885a06f1-aaad-470e-b98a-3c11567fb498
14052024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_277c7b01-7629-4266-a541-a595a35b33f0
14062024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_pantry_a92c41e2-55d8-410d-a76b-75046488191b
14072024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_oximeter_af0a89d1-a224-45ab-9675-7089dedc5fe9
14082024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_clickhouse_b9560b7f-62aa-42db-a117-057b548be234
14092024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_b24c01d6-5b69-4c4d-87c7-e1ebe641dcad
14102024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_nexus_221b6ce2-61f2-4949-9537-b4af13ae839b
14112024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_5b488837-4873-4ac9-bd84-9bee89316c41
14122024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_pantry_a9a229b2-32e9-4ad4-8b94-5a16ef0326c9
14132024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_crucible_7d304db3-3ef6-470f-8340-a6fdffd8f677
14142024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_nexus_09df608b-9da6-413f-9a4b-1808bcd7a800
14152024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_external_dns_effcf430-6675-455f-820f-495829fc7071
14162024-06-28T22:10:06.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2966 zone = oxz_nexus_88974d37-08ca-4371-bfcc-d8e58336bbf5
14172024-06-28T22:10:06.816ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
14182024-06-28T22:10:06.824ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
14192024-06-28T22:10:06.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 33134800 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53048 req_id = 6dfacbfa-7dd8-4fed-85b6-80a8f08a2866 response_code = 204 uri = /omicron-zones
14202024-06-28T22:10:06.857ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1244
14212024-06-28T22:10:06.864ZINFOSledAgent (RSS): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
14222024-06-28T22:10:06.865ZINFOSledAgent (RSS): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
14232024-06-28T22:10:06.873ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:681
14242024-06-28T22:10:06.888ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:694
14252024-06-28T22:10:06.929ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:825
14262024-06-28T22:10:07.210ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.191125029s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:423 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
14272024-06-28T22:10:08.126ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14282024-06-28T22:10:08.221ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14292024-06-28T22:10:08.301ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14302024-06-28T22:10:08.500ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14312024-06-28T22:10:08.634ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14322024-06-28T22:10:08.883ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14332024-06-28T22:10:09.255ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14342024-06-28T22:10:09.432ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 11.232126718s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:423 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
14352024-06-28T22:10:09.576ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14362024-06-28T22:10:09.964ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14372024-06-28T22:10:10.512ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14382024-06-28T22:10:10.850ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14392024-06-28T22:10:11.252ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14402024-06-28T22:10:12.018ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14412024-06-28T22:10:12.653ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14422024-06-28T22:10:13.569ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14432024-06-28T22:10:14.519ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14442024-06-28T22:10:15.653ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/22728d30-b15c-4a22-83e6-f1b1e01718ef/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
14452024-06-28T22:10:16.002ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51337
14462024-06-28T22:10:16.004ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 678 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51337 req_id = 4d65bf00-76d4-48f3-a9a6-e16d68f8df37 response_code = 200 uri = /baseboard
14472024-06-28T22:10:18.275ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 403 Forbidden; headers: {"content-type": "application/json", "x-request-id": "cb718400-c1c7-4b3d-8de5-8c3905e81262", "content-length": "113", "date": "Fri, 28 Jun 2024 22:10:18 GMT"}; value: Error { error_code: Some("Forbidden"), message: "Forbidden", request_id: "cb718400-c1c7-4b3d-8de5-8c3905e81262" } file = sled-agent/src/rack_setup/service.rs:887
14482024-06-28T22:10:20.845ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "751b4f0f-c635-4fff-80fd-ef77eea646fe", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:20 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "751b4f0f-c635-4fff-80fd-ef77eea646fe" } file = sled-agent/src/rack_setup/service.rs:887
14492024-06-28T22:10:22.580ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "953d4110-1e85-4be4-8f15-68ba4f931156", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:22 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "953d4110-1e85-4be4-8f15-68ba4f931156" } file = sled-agent/src/rack_setup/service.rs:887
14502024-06-28T22:10:24.820ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "cc2a9e4c-dd04-41cf-b339-2a0f686ebbf5", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:24 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "cc2a9e4c-dd04-41cf-b339-2a0f686ebbf5" } file = sled-agent/src/rack_setup/service.rs:887
14512024-06-28T22:10:27.252ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "3ee56b37-4ba8-4d33-a670-f02c21a7cac1", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:27 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "3ee56b37-4ba8-4d33-a670-f02c21a7cac1" } file = sled-agent/src/rack_setup/service.rs:887
14522024-06-28T22:10:31.275ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "668e083f-dc32-4fdb-99b4-69e7f033c55e", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:30 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "668e083f-dc32-4fdb-99b4-69e7f033c55e" } file = sled-agent/src/rack_setup/service.rs:887
14532024-06-28T22:10:35.799ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "95b60b22-ab95-4078-9ddb-404006189391", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:35 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "95b60b22-ab95-4078-9ddb-404006189391" } file = sled-agent/src/rack_setup/service.rs:887
14542024-06-28T22:10:45.260ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "7c08959d-9a74-4375-b000-2d43a4fda8a0", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:44 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "7c08959d-9a74-4375-b000-2d43a4fda8a0" } file = sled-agent/src/rack_setup/service.rs:887
14552024-06-28T22:10:46.042ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34753
14562024-06-28T22:10:46.042ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 29 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34753 req_id = 14ce43c7-c7b3-458d-bdab-e012efe419b2 response_code = 200 uri = /baseboard
14572024-06-28T22:10:50.817ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:41782 remote_addr = [fd00:1122:3344:101::d]:65345 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
14582024-06-28T22:10:50.826ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7959 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = b0d82179-d692-43a5-9fa8-49e6ce3a4a3c response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
14592024-06-28T22:10:52.261ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "b5587a56-2480-4899-9c3b-54539d55676b", "content-length": "173", "date": "Fri, 28 Jun 2024 22:10:52 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "b5587a56-2480-4899-9c3b-54539d55676b" } file = sled-agent/src/rack_setup/service.rs:887
14602024-06-28T22:11:00.776ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "c3bef75b-f1af-4cfb-a676-4a4cba977193", "content-length": "173", "date": "Fri, 28 Jun 2024 22:11:00 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a9021c63-7f96-431e-9198-9a7654ae0f67\"", request_id: "c3bef75b-f1af-4cfb-a676-4a4cba977193" } file = sled-agent/src/rack_setup/service.rs:887
14612024-06-28T22:11:01.088ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67
14622024-06-28T22:11:01.176ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57924
14632024-06-28T22:11:01.177ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14642024-06-28T22:11:01.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 87614 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57924 req_id = 9fea54da-e031-4a1a-a06a-65ba2451fead response_code = 200 uri = /inventory
14652024-06-28T22:11:01.265ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14662024-06-28T22:11:01.265ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-zones.json file = common/src/ledger.rs:177
14672024-06-28T22:11:01.267ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-zones.json file = common/src/ledger.rs:177
14682024-06-28T22:11:01.267ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3291 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57924 req_id = 9dc5a7d5-09da-4266-b1f3-18bf36da1022 response_code = 200 uri = /omicron-zones
14692024-06-28T22:11:12.865ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53488
14702024-06-28T22:11:12.870ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:690 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5ab9e19a-f9b8-42cb-b346-aff1c689c92e, kind: Service { id: a52585f4-7208-41c3-ac37-3029cbd9b012 }, name: Name("external-dns-a52585f4-7208-41c3-ac37-3029cbd9b012"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: a20b9ed9-42a5-4070-ba7b-11203172977a, kind: Service { id: effcf430-6675-455f-820f-495829fc7071 }, name: Name("external-dns-effcf430-6675-455f-820f-495829fc7071"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 155])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 35808902-45c0-47d5-8544-cb7fdaca7e8c, kind: Service { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5 }, name: Name("nexus-88974d37-08ca-4371-bfcc-d8e58336bbf5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 69])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ba613ac1-2766-48e3-acdb-eb29ca6e137f, kind: Service { id: 09df608b-9da6-413f-9a4b-1808bcd7a800 }, name: Name("nexus-09df608b-9da6-413f-9a4b-1808bcd7a800"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 109])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f475e35c-414a-4bcc-b0ee-9464f4e4dbaf, kind: Service { id: 221b6ce2-61f2-4949-9537-b4af13ae839b }, name: Name("nexus-221b6ce2-61f2-4949-9537-b4af13ae839b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 119])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14712024-06-28T22:11:12.871ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte0" rules = []
14722024-06-28T22:11:12.871ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14732024-06-28T22:11:12.871ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14742024-06-28T22:11:12.871ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14752024-06-28T22:11:12.871ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14762024-06-28T22:11:12.871ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14772024-06-28T22:11:12.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 5984 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53488 req_id = 5ab8dfac-b137-4c8d-ab1d-ca5262e875bb response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14782024-06-28T22:11:12.872ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:897
14792024-06-28T22:11:13.009ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49039
14802024-06-28T22:11:13.009ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14812024-06-28T22:11:13.098ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 89090 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49039 req_id = 8c7ae88c-3c22-4dd8-bc81-7915a802b700 response_code = 200 uri = /inventory
14822024-06-28T22:11:13.099ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14832024-06-28T22:11:13.099ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-zones.json file = common/src/ledger.rs:177
14842024-06-28T22:11:13.099ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-zones.json file = common/src/ledger.rs:177
14852024-06-28T22:11:13.099ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 716 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49039 req_id = 3881bd0d-a5bc-441b-b6b2-848f32a487c0 response_code = 200 uri = /omicron-zones
14862024-06-28T22:11:14.459ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54202
14872024-06-28T22:11:14.459ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:690 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5ab9e19a-f9b8-42cb-b346-aff1c689c92e, kind: Service { id: a52585f4-7208-41c3-ac37-3029cbd9b012 }, name: Name("external-dns-a52585f4-7208-41c3-ac37-3029cbd9b012"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: a20b9ed9-42a5-4070-ba7b-11203172977a, kind: Service { id: effcf430-6675-455f-820f-495829fc7071 }, name: Name("external-dns-effcf430-6675-455f-820f-495829fc7071"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 155])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 35808902-45c0-47d5-8544-cb7fdaca7e8c, kind: Service { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5 }, name: Name("nexus-88974d37-08ca-4371-bfcc-d8e58336bbf5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 69])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ba613ac1-2766-48e3-acdb-eb29ca6e137f, kind: Service { id: 09df608b-9da6-413f-9a4b-1808bcd7a800 }, name: Name("nexus-09df608b-9da6-413f-9a4b-1808bcd7a800"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 109])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f475e35c-414a-4bcc-b0ee-9464f4e4dbaf, kind: Service { id: 221b6ce2-61f2-4949-9537-b4af13ae839b }, name: Name("nexus-221b6ce2-61f2-4949-9537-b4af13ae839b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 119])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14882024-06-28T22:11:14.459ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte0" rules = []
14892024-06-28T22:11:14.459ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14902024-06-28T22:11:14.459ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14912024-06-28T22:11:14.459ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14922024-06-28T22:11:14.459ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14932024-06-28T22:11:14.459ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14942024-06-28T22:11:14.459ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 252 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54202 req_id = ed7b78af-56b9-4dd5-ad86-baeca4933588 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14952024-06-28T22:11:14.512ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41240
14962024-06-28T22:11:14.512ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:690 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5ab9e19a-f9b8-42cb-b346-aff1c689c92e, kind: Service { id: a52585f4-7208-41c3-ac37-3029cbd9b012 }, name: Name("external-dns-a52585f4-7208-41c3-ac37-3029cbd9b012"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: a20b9ed9-42a5-4070-ba7b-11203172977a, kind: Service { id: effcf430-6675-455f-820f-495829fc7071 }, name: Name("external-dns-effcf430-6675-455f-820f-495829fc7071"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 155])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 35808902-45c0-47d5-8544-cb7fdaca7e8c, kind: Service { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5 }, name: Name("nexus-88974d37-08ca-4371-bfcc-d8e58336bbf5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 69])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ba613ac1-2766-48e3-acdb-eb29ca6e137f, kind: Service { id: 09df608b-9da6-413f-9a4b-1808bcd7a800 }, name: Name("nexus-09df608b-9da6-413f-9a4b-1808bcd7a800"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 109])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f475e35c-414a-4bcc-b0ee-9464f4e4dbaf, kind: Service { id: 221b6ce2-61f2-4949-9537-b4af13ae839b }, name: Name("nexus-221b6ce2-61f2-4949-9537-b4af13ae839b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 119])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14972024-06-28T22:11:14.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte0" rules = []
14982024-06-28T22:11:14.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14992024-06-28T22:11:14.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15002024-06-28T22:11:14.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15012024-06-28T22:11:14.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15022024-06-28T22:11:14.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15032024-06-28T22:11:14.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41240 req_id = e87eb573-f7aa-414e-af8c-008efe9aee6f response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15042024-06-28T22:11:14.514ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53108
15052024-06-28T22:11:14.514ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:690 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5ab9e19a-f9b8-42cb-b346-aff1c689c92e, kind: Service { id: a52585f4-7208-41c3-ac37-3029cbd9b012 }, name: Name("external-dns-a52585f4-7208-41c3-ac37-3029cbd9b012"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: a20b9ed9-42a5-4070-ba7b-11203172977a, kind: Service { id: effcf430-6675-455f-820f-495829fc7071 }, name: Name("external-dns-effcf430-6675-455f-820f-495829fc7071"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 155])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 35808902-45c0-47d5-8544-cb7fdaca7e8c, kind: Service { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5 }, name: Name("nexus-88974d37-08ca-4371-bfcc-d8e58336bbf5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 69])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ba613ac1-2766-48e3-acdb-eb29ca6e137f, kind: Service { id: 09df608b-9da6-413f-9a4b-1808bcd7a800 }, name: Name("nexus-09df608b-9da6-413f-9a4b-1808bcd7a800"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 109])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f475e35c-414a-4bcc-b0ee-9464f4e4dbaf, kind: Service { id: 221b6ce2-61f2-4949-9537-b4af13ae839b }, name: Name("nexus-221b6ce2-61f2-4949-9537-b4af13ae839b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 119])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15062024-06-28T22:11:14.514ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte0" rules = []
15072024-06-28T22:11:14.514ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15082024-06-28T22:11:14.514ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15092024-06-28T22:11:14.514ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15102024-06-28T22:11:14.514ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15112024-06-28T22:11:14.514ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15122024-06-28T22:11:14.514ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 180 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53108 req_id = 442e9387-9a33-4abb-93d5-abfc293da4a7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15132024-06-28T22:11:16.073ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41504
15142024-06-28T22:11:16.073ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 62 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41504 req_id = fe77faa4-e2ff-48c7-b998-d4439ac7e25f response_code = 200 uri = /baseboard
15152024-06-28T22:11:19.426ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37712
15162024-06-28T22:11:19.427ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 779 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37712 req_id = 6e599d98-8afc-4d76-9fc7-a857a067efd4 response_code = 200 uri = /v2p
15172024-06-28T22:11:19.490ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62812
15182024-06-28T22:11:19.490ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 471 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62812 req_id = 2148e6ea-d1dd-4281-8a1e-33ec06194055 response_code = 200 uri = /vpc-routes
15192024-06-28T22:11:19.557ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 4048 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62812 req_id = 1a80a434-c282-4f7c-82a9-df7a38140fea response_code = 204 uri = /vpc-routes
15202024-06-28T22:11:20.042ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56192
15212024-06-28T22:11:20.139ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97390 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56192 req_id = eb0b3b89-4694-403a-899a-d1ecfd9f95e1 response_code = 204 uri = /switch-ports
15222024-06-28T22:11:20.142ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 2139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56192 req_id = 4c618d66-451b-48e3-ada0-bfbda519638e response_code = 200 uri = /network-bootstore-config
15232024-06-28T22:11:20.170ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
15242024-06-28T22:11:20.171ZINFOSledAgent (bootstore): Writing ledger to /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
15252024-06-28T22:11:20.172ZINFOSledAgent (bootstore): Writing ledger to /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
15262024-06-28T22:11:20.173ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
15272024-06-28T22:11:20.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 4039 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56192 req_id = 03373521-39ce-41a4-9c56-dc933b44f141 response_code = 204 uri = /network-bootstore-config
15282024-06-28T22:11:20.817ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1260 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = eeb930c9-ce05-4b64-bb6b-fa02f8a1d7b6 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
15292024-06-28T22:11:46.109ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48825
15302024-06-28T22:11:46.109ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 68 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48825 req_id = 89c652f1-72d3-4bf0-b5fd-d0b7a231ab0f response_code = 200 uri = /baseboard
15312024-06-28T22:11:49.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37744
15322024-06-28T22:11:49.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 739 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37744 req_id = 97f38a5e-258b-46b5-b836-556080b6308c response_code = 200 uri = /vpc-routes
15332024-06-28T22:11:49.416ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41565
15342024-06-28T22:11:49.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 321 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41565 req_id = 4688db46-85d4-42ce-bd4a-fc5a1390cb4c response_code = 200 uri = /v2p
15352024-06-28T22:11:49.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37744 req_id = dcc3e7a4-53dc-4fbc-a2cc-0c553fae983d response_code = 204 uri = /vpc-routes
15362024-06-28T22:11:49.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35866
15372024-06-28T22:11:49.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44287 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35866 req_id = 40c91bfa-f0c4-481f-93bd-80efeb5cd441 response_code = 204 uri = /switch-ports
15382024-06-28T22:11:49.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35866 req_id = f091aad5-dd3b-430a-af25-72e908041133 response_code = 200 uri = /network-bootstore-config
15392024-06-28T22:11:50.821ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1130 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = cfa1e3c5-d96b-4770-97bc-3cb65a75f515 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
15402024-06-28T22:12:16.145ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52178
15412024-06-28T22:12:16.145ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52178 req_id = 7b6411b2-b7b9-4957-8178-1ca79cc03e56 response_code = 200 uri = /baseboard
15422024-06-28T22:12:19.336ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44983
15432024-06-28T22:12:19.336ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 189 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44983 req_id = 3de2b6d8-5007-4e0b-8985-330234651486 response_code = 200 uri = /v2p
15442024-06-28T22:12:19.439ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53939
15452024-06-28T22:12:19.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53939 req_id = 9447f1e6-1d5c-4fc8-84d0-69311ded7d52 response_code = 200 uri = /vpc-routes
15462024-06-28T22:12:19.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1046 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53939 req_id = 7af2b7f9-a58a-483f-bbe2-9b4c0589627b response_code = 204 uri = /vpc-routes
15472024-06-28T22:12:19.650ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32927
15482024-06-28T22:12:19.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 42370 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:32927 req_id = 737a09de-0676-4fe3-9a52-6a9846b2268f response_code = 204 uri = /switch-ports
15492024-06-28T22:12:19.693ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32927 req_id = 5610bb4d-17cc-4a14-9906-9f774be4c37a response_code = 200 uri = /network-bootstore-config
15502024-06-28T22:12:20.826ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 2192 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = 5f0504e4-8d32-4055-a461-c3d44d6d349b response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
15512024-06-28T22:12:46.179ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53962
15522024-06-28T22:12:46.179ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 117 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53962 req_id = 5baa6f6d-1012-410e-bb6c-f18ad9cbe218 response_code = 200 uri = /baseboard
15532024-06-28T22:12:49.339ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35392
15542024-06-28T22:12:49.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35392 req_id = 88afce8b-1bb5-47b0-a985-c46c20d38a56 response_code = 200 uri = /vpc-routes
15552024-06-28T22:12:49.346ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 778 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35392 req_id = deccb25d-7a4e-4ff5-80c1-d468cd65424a response_code = 204 uri = /vpc-routes
15562024-06-28T22:12:49.355ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59103
15572024-06-28T22:12:49.355ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59103 req_id = efe72656-7a3c-414c-878e-b7a5fcc374b6 response_code = 200 uri = /v2p
15582024-06-28T22:12:49.538ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36996
15592024-06-28T22:12:49.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46425 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36996 req_id = 9593ff00-ef6b-48c6-b6a9-7c4e3a16dabd response_code = 204 uri = /switch-ports
15602024-06-28T22:12:49.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36996 req_id = 18b05d22-22ed-43fb-84df-32fe1e66a157 response_code = 200 uri = /network-bootstore-config
15612024-06-28T22:12:50.836ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 318 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = db23539c-51e8-440a-b501-6037f61e106f response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
15622024-06-28T22:13:16.212ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54043
15632024-06-28T22:13:16.213ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 83 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54043 req_id = 8d342cb2-1163-4b8e-820e-a674963653c3 response_code = 200 uri = /baseboard
15642024-06-28T22:13:19.439ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62625
15652024-06-28T22:13:19.440ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62625 req_id = 74fd201e-5bed-4018-b4ed-06bc19d2b9fe response_code = 200 uri = /v2p
15662024-06-28T22:13:19.462ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49781
15672024-06-28T22:13:19.463ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49781 req_id = 617ab7c9-b0a1-441f-bf13-93f375f5c013 response_code = 200 uri = /vpc-routes
15682024-06-28T22:13:19.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49781 req_id = a8d228eb-5459-43d9-8068-97f15ef80f5d response_code = 204 uri = /vpc-routes
15692024-06-28T22:13:19.705ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38350
15702024-06-28T22:13:19.749ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 43969 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38350 req_id = df4e799f-77e1-4efd-a66c-a207a2b4d7a9 response_code = 204 uri = /switch-ports
15712024-06-28T22:13:19.750ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38350 req_id = 3139cd53-7d9c-4e7b-a749-58cc967e9370 response_code = 200 uri = /network-bootstore-config
15722024-06-28T22:13:20.878ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 587 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = b7234440-51ce-4706-8957-e78285d6f26f response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
15732024-06-28T22:13:32.845ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
15742024-06-28T22:13:46.246ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:32893
15752024-06-28T22:13:46.247ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 87 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:32893 req_id = b6e283fd-5971-4c97-9bce-6d18988ce373 response_code = 200 uri = /baseboard
15762024-06-28T22:13:49.442ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60285
15772024-06-28T22:13:49.443ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 634 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60285 req_id = e44f96be-b27f-4489-9910-b9c6df338342 response_code = 200 uri = /vpc-routes
15782024-06-28T22:13:49.449ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60285 req_id = e6af5900-5199-416b-91e0-5112abc00da7 response_code = 204 uri = /vpc-routes
15792024-06-28T22:13:49.451ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41057
15802024-06-28T22:13:49.451ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41057 req_id = 2723e970-960a-4fdf-bd1f-20765f2c8ef8 response_code = 200 uri = /v2p
15812024-06-28T22:13:49.502ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52126
15822024-06-28T22:13:49.545ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 43405 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52126 req_id = ac8d8511-d831-40ec-8363-6208ab80ffd6 response_code = 204 uri = /switch-ports
15832024-06-28T22:13:49.546ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 834 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52126 req_id = 49217fd7-4790-4a62-92f5-15a2c7fded44 response_code = 200 uri = /network-bootstore-config
15842024-06-28T22:13:50.880ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1087 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = e473aa10-318e-408a-bd20-5d9c138762e4 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
15852024-06-28T22:14:16.277ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36501
15862024-06-28T22:14:16.278ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 106 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36501 req_id = f77c22bf-6dec-44f0-956f-df7f8ff2922a response_code = 200 uri = /baseboard
15872024-06-28T22:14:19.339ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58475
15882024-06-28T22:14:19.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58475 req_id = e8efa820-c085-4d5c-82a3-c207736e9122 response_code = 200 uri = /vpc-routes
15892024-06-28T22:14:19.345ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58475 req_id = 9cf72e33-7863-4c0d-8408-bfed9c1617ed response_code = 204 uri = /vpc-routes
15902024-06-28T22:14:19.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57861
15912024-06-28T22:14:19.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57861 req_id = 67f81e69-8334-406c-a0c0-b5e5351bff16 response_code = 200 uri = /v2p
15922024-06-28T22:14:19.734ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46406
15932024-06-28T22:14:19.778ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44728 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46406 req_id = 18969508-bf8f-4163-844a-615180772349 response_code = 204 uri = /switch-ports
15942024-06-28T22:14:19.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 701 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46406 req_id = b0c601f9-8ebd-4c2e-af44-f3fb57e2c9c8 response_code = 200 uri = /network-bootstore-config
15952024-06-28T22:14:20.881ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 629 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = 8d5ec86b-7a26-4ed9-96d4-ad0271745fec response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
15962024-06-28T22:14:46.311ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36812
15972024-06-28T22:14:46.311ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 91 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36812 req_id = d31f3205-72b7-42c6-9ce8-ae1189aba0a5 response_code = 200 uri = /baseboard
15982024-06-28T22:14:49.272ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50065
15992024-06-28T22:14:49.272ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50065 req_id = 58487ee0-5029-4568-bdc5-620bf84481f4 response_code = 200 uri = /vpc-routes
16002024-06-28T22:14:49.278ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50065 req_id = 1dbd9c98-906f-4559-980a-ccede212e6db response_code = 204 uri = /vpc-routes
16012024-06-28T22:14:49.325ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64685
16022024-06-28T22:14:49.325ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64685 req_id = d4419f92-cafe-4b2a-882e-27cd50f4022f response_code = 200 uri = /v2p
16032024-06-28T22:14:49.392ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41311
16042024-06-28T22:14:49.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47432 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:41311 req_id = 280d8966-2e04-45e2-a2d8-ef96be2d49d1 response_code = 204 uri = /switch-ports
16052024-06-28T22:14:49.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41311 req_id = 1c5a9406-0f21-4f19-ba4f-ed1f3b069849 response_code = 200 uri = /network-bootstore-config
16062024-06-28T22:14:50.882ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 838 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = 84a13cd0-d33a-4275-9c16-2f7adf0d4d26 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16072024-06-28T22:15:16.343ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55111
16082024-06-28T22:15:16.343ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 129 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55111 req_id = c86939a0-2b45-4546-a664-a1b24ecf8abc response_code = 200 uri = /baseboard
16092024-06-28T22:15:19.276ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64564
16102024-06-28T22:15:19.276ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64564 req_id = b717c948-f0f1-4ce9-8927-9da1f2c70d10 response_code = 200 uri = /vpc-routes
16112024-06-28T22:15:19.281ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64564 req_id = e494947c-876a-424c-a78a-e7114df9ce56 response_code = 204 uri = /vpc-routes
16122024-06-28T22:15:19.321ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65166
16132024-06-28T22:15:19.321ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65166 req_id = 002372b5-915c-473b-a51d-f0d6f7918c8a response_code = 200 uri = /v2p
16142024-06-28T22:15:19.498ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43446
16152024-06-28T22:15:19.498ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:690 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5ab9e19a-f9b8-42cb-b346-aff1c689c92e, kind: Service { id: a52585f4-7208-41c3-ac37-3029cbd9b012 }, name: Name("external-dns-a52585f4-7208-41c3-ac37-3029cbd9b012"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: a20b9ed9-42a5-4070-ba7b-11203172977a, kind: Service { id: effcf430-6675-455f-820f-495829fc7071 }, name: Name("external-dns-effcf430-6675-455f-820f-495829fc7071"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 155])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 35808902-45c0-47d5-8544-cb7fdaca7e8c, kind: Service { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5 }, name: Name("nexus-88974d37-08ca-4371-bfcc-d8e58336bbf5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 69])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ba613ac1-2766-48e3-acdb-eb29ca6e137f, kind: Service { id: 09df608b-9da6-413f-9a4b-1808bcd7a800 }, name: Name("nexus-09df608b-9da6-413f-9a4b-1808bcd7a800"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 109])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f475e35c-414a-4bcc-b0ee-9464f4e4dbaf, kind: Service { id: 221b6ce2-61f2-4949-9537-b4af13ae839b }, name: Name("nexus-221b6ce2-61f2-4949-9537-b4af13ae839b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 119])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16162024-06-28T22:15:19.498ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte0" rules = []
16172024-06-28T22:15:19.498ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16182024-06-28T22:15:19.498ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16192024-06-28T22:15:19.498ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16202024-06-28T22:15:19.498ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16212024-06-28T22:15:19.498ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16222024-06-28T22:15:19.498ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 321 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43446 req_id = cbdf3179-56b6-42f6-87d0-6e1d1731d828 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16232024-06-28T22:15:19.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47880
16242024-06-28T22:15:19.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 61658 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:47880 req_id = e2579885-6c9b-47f5-aff9-74a5240be646 response_code = 204 uri = /switch-ports
16252024-06-28T22:15:19.954ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47880 req_id = 022a2661-74f7-4dc3-b20d-75876b05f6f8 response_code = 200 uri = /network-bootstore-config
16262024-06-28T22:15:20.882ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 626 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = d98b39be-a1ef-4ba6-952c-4ac4eeb0b1a3 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16272024-06-28T22:15:46.375ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35080
16282024-06-28T22:15:46.375ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35080 req_id = 5383e713-f0a0-4e2d-a8ee-e35e730c7e69 response_code = 200 uri = /baseboard
16292024-06-28T22:15:49.233ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64948
16302024-06-28T22:15:49.233ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64948 req_id = da3b54d7-477d-45cd-bd29-2ed4ee7250b6 response_code = 200 uri = /vpc-routes
16312024-06-28T22:15:49.239ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64948 req_id = 525d2447-2a83-4560-a9da-c889a73f4502 response_code = 204 uri = /vpc-routes
16322024-06-28T22:15:49.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60256
16332024-06-28T22:15:49.256ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60256 req_id = a6954d5a-52ee-442c-a888-16a4b2054a87 response_code = 200 uri = /v2p
16342024-06-28T22:15:49.436ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38509
16352024-06-28T22:15:49.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 42388 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38509 req_id = 212b2af0-914b-4f08-a6f6-a73f570dcdd3 response_code = 204 uri = /switch-ports
16362024-06-28T22:15:49.479ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 758 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38509 req_id = 4527434e-9e26-4a03-b170-0ef1886a58b0 response_code = 200 uri = /network-bootstore-config
16372024-06-28T22:15:50.969ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 703 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = b05f29c4-3ce6-41d9-ac9f-f644f5de3c12 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16382024-06-28T22:16:16.406ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46456
16392024-06-28T22:16:16.406ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 74 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46456 req_id = 234582d6-c95e-4b2a-8811-f140f926a821 response_code = 200 uri = /baseboard
16402024-06-28T22:16:19.355ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33941
16412024-06-28T22:16:19.355ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33941 req_id = 527dafc7-7ec2-4ca7-8668-e9edca7cc207 response_code = 200 uri = /vpc-routes
16422024-06-28T22:16:19.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 875 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33941 req_id = d9c3c326-90a1-49cd-90b2-cfd12787a22d response_code = 204 uri = /vpc-routes
16432024-06-28T22:16:19.490ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60834
16442024-06-28T22:16:19.491ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60834 req_id = a7e563e7-8336-4be0-bda0-9561f454c919 response_code = 200 uri = /v2p
16452024-06-28T22:16:19.837ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39326
16462024-06-28T22:16:19.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 42299 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39326 req_id = dd91abed-284e-4c8a-af73-606879fda842 response_code = 204 uri = /switch-ports
16472024-06-28T22:16:19.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39326 req_id = 303eb1bf-da53-49e0-a99c-2a40868e0444 response_code = 200 uri = /network-bootstore-config
16482024-06-28T22:16:20.969ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 857 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = 0a1eb5f2-8ae8-4574-ba11-5db75c26adf8 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16492024-06-28T22:16:46.439ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38069
16502024-06-28T22:16:46.440ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 143 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38069 req_id = 0ad2c4af-12b6-47bc-84e0-6cb962986584 response_code = 200 uri = /baseboard
16512024-06-28T22:16:49.273ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36479
16522024-06-28T22:16:49.274ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36479 req_id = 25207fda-0b72-454f-b209-332547904bd1 response_code = 200 uri = /v2p
16532024-06-28T22:16:49.360ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62952
16542024-06-28T22:16:49.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62952 req_id = 38bbe2ac-675c-470d-a690-09ec13962cab response_code = 200 uri = /vpc-routes
16552024-06-28T22:16:49.366ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62952 req_id = 9457d3ed-0376-409c-b3e8-ef8f5761107b response_code = 204 uri = /vpc-routes
16562024-06-28T22:16:49.482ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33931
16572024-06-28T22:16:49.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45987 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33931 req_id = bbc64b2e-5dd9-4f87-b8e5-ba5c9c38966b response_code = 204 uri = /switch-ports
16582024-06-28T22:16:49.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33931 req_id = 7e365bf2-49f0-46d8-9c59-ead3fae4a40b response_code = 200 uri = /network-bootstore-config
16592024-06-28T22:16:50.974ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 549 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = 637ce109-5e1d-431b-923f-61924924a543 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16602024-06-28T22:17:16.471ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61733
16612024-06-28T22:17:16.471ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 77 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61733 req_id = 828552d2-43dc-447a-8706-cdadc54f9d3e response_code = 200 uri = /baseboard
16622024-06-28T22:17:19.280ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55526
16632024-06-28T22:17:19.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 801 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55526 req_id = fa288110-fdd3-4547-a002-ae48e448d5a2 response_code = 200 uri = /v2p
16642024-06-28T22:17:19.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33893
16652024-06-28T22:17:19.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33893 req_id = 5aa1c159-099b-4c08-ba7b-b13dd7bf6132 response_code = 200 uri = /vpc-routes
16662024-06-28T22:17:19.539ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 519 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33893 req_id = b344d826-ae22-40d9-a1d6-831d8bd6ada6 response_code = 204 uri = /vpc-routes
16672024-06-28T22:17:19.645ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37934
16682024-06-28T22:17:19.690ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44732 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37934 req_id = 5f6b306b-b75e-4b53-827a-6884560df824 response_code = 204 uri = /switch-ports
16692024-06-28T22:17:19.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 805 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37934 req_id = 3266607c-75f4-4c9b-b704-004de7fbcdbd response_code = 200 uri = /network-bootstore-config
16702024-06-28T22:17:20.970ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 661 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = acab0279-1f31-4b90-8e60-e5a75bc1ddee response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16712024-06-28T22:17:46.504ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38078
16722024-06-28T22:17:46.504ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38078 req_id = 12ccf019-0887-4c55-a806-74e55bef22d2 response_code = 200 uri = /baseboard
16732024-06-28T22:17:49.221ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62345
16742024-06-28T22:17:49.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62345 req_id = a1236e1c-c9a6-42cb-8b7c-2d06d88b6389 response_code = 200 uri = /vpc-routes
16752024-06-28T22:17:49.227ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62345 req_id = 50e3a5f1-b6da-4a5a-aad8-f4d98385bcfe response_code = 204 uri = /vpc-routes
16762024-06-28T22:17:49.240ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40830
16772024-06-28T22:17:49.240ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40830 req_id = c481f5ff-f0ee-49d9-ba55-5a845d2dc994 response_code = 200 uri = /v2p
16782024-06-28T22:17:49.440ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45742
16792024-06-28T22:17:49.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45336 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:45742 req_id = 6b875a06-cb66-4d5c-b56d-36bd47420274 response_code = 204 uri = /switch-ports
16802024-06-28T22:17:49.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45742 req_id = 3f7d181a-860d-4505-b0fa-710d0d0e24ac response_code = 200 uri = /network-bootstore-config
16812024-06-28T22:17:50.970ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 888 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = ed1bb624-d00c-4910-8e4f-b5dbddf8597a response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16822024-06-28T22:18:16.536ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58331
16832024-06-28T22:18:16.536ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 71 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58331 req_id = af6888e8-6337-400d-9e20-5207667d98e9 response_code = 200 uri = /baseboard
16842024-06-28T22:18:19.190ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:300
16852024-06-28T22:18:19.190ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16862024-06-28T22:18:19.228ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 2148, tv_nsec: 937090553 } time_to_next_cleanup = 599.99999677s
16872024-06-28T22:18:19.308ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58709
16882024-06-28T22:18:19.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58709 req_id = de4890e8-4e9e-4b57-9995-4d2b10074662 response_code = 200 uri = /v2p
16892024-06-28T22:18:19.432ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46045
16902024-06-28T22:18:19.433ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 826 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46045 req_id = 0962026e-1a2d-4b93-a2dd-edd7469923f1 response_code = 200 uri = /vpc-routes
16912024-06-28T22:18:19.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46045 req_id = e92bbed8-0ee2-4b15-b27b-3f9e70a3d9fa response_code = 204 uri = /vpc-routes
16922024-06-28T22:18:19.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36835
16932024-06-28T22:18:19.654ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45203 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36835 req_id = 7fcee1a1-b028-4301-b4fc-04a8f8d32c58 response_code = 204 uri = /switch-ports
16942024-06-28T22:18:19.654ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36835 req_id = 1ede5027-99c1-4cb4-8ab3-3df6f6dd7617 response_code = 200 uri = /network-bootstore-config
16952024-06-28T22:18:21.048ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 537 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = d960b4a1-a66f-4bdd-a2b7-294a46456969 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
16962024-06-28T22:18:32.851ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
16972024-06-28T22:18:46.569ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40191
16982024-06-28T22:18:46.569ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 88 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40191 req_id = 63edc260-6254-4707-8db9-bcd3e27737fa response_code = 200 uri = /baseboard
16992024-06-28T22:18:49.230ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65463
17002024-06-28T22:18:49.230ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65463 req_id = f3e9ea31-b8e3-49b8-9b26-2e6665d25ad0 response_code = 200 uri = /vpc-routes
17012024-06-28T22:18:49.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:65463 req_id = 972c6620-e4c6-404e-99e9-1db8753e874d response_code = 204 uri = /vpc-routes
17022024-06-28T22:18:49.256ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48878
17032024-06-28T22:18:49.257ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 705 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48878 req_id = 06988217-1f9f-40fd-b860-c9c14d73b742 response_code = 200 uri = /v2p
17042024-06-28T22:18:49.481ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65192
17052024-06-28T22:18:49.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45554 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:65192 req_id = bf8369af-2ee8-4539-a602-192c2bbf626d response_code = 204 uri = /switch-ports
17062024-06-28T22:18:49.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65192 req_id = 93b8f47b-2a44-4676-8bef-c7d8e2b93445 response_code = 200 uri = /network-bootstore-config
17072024-06-28T22:18:51.049ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1466 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = 820d67e9-aafb-4dc3-81da-54327ec2a7a7 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
17082024-06-28T22:19:16.600ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58233
17092024-06-28T22:19:16.601ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 72 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58233 req_id = 3160c767-e051-409b-a33c-5663c1b53a7f response_code = 200 uri = /baseboard
17102024-06-28T22:19:19.420ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58473
17112024-06-28T22:19:19.421ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 607 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58473 req_id = 0b59c666-8331-4bc0-9340-42cc27c50e70 response_code = 200 uri = /vpc-routes
17122024-06-28T22:19:19.424ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63056
17132024-06-28T22:19:19.424ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63056 req_id = c83755e5-4e74-4cd7-a7be-17ee004a466c response_code = 200 uri = /v2p
17142024-06-28T22:19:19.426ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 236 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58473 req_id = aad7a3dd-20da-4909-96a4-9b1a5f34af05 response_code = 204 uri = /vpc-routes
17152024-06-28T22:19:19.626ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54127
17162024-06-28T22:19:19.671ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45096 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54127 req_id = d0c3eb2f-8cf8-4079-a3a8-7581e1f9631a response_code = 204 uri = /switch-ports
17172024-06-28T22:19:19.671ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54127 req_id = 56ac6fbc-e8b1-4213-8e97-cd721dd23ef2 response_code = 200 uri = /network-bootstore-config
17182024-06-28T22:19:21.049ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1085 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = e2bb0de7-ddf0-46cd-a8c6-e76b4cf59a53 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
17192024-06-28T22:19:46.631ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60011
17202024-06-28T22:19:46.631ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 109 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60011 req_id = c45268b7-9dbf-407e-850c-0b05d3984957 response_code = 200 uri = /baseboard
17212024-06-28T22:19:49.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59570
17222024-06-28T22:19:49.255ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59570 req_id = 00fed1fa-14ad-4805-ad63-8082821d1101 response_code = 200 uri = /vpc-routes
17232024-06-28T22:19:49.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59570 req_id = 4f2371e6-8017-4b9d-bcc8-fb1415271cbc response_code = 204 uri = /vpc-routes
17242024-06-28T22:19:49.269ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38141
17252024-06-28T22:19:49.269ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38141 req_id = 4698529b-d52d-4157-9b12-49dc9f489611 response_code = 200 uri = /v2p
17262024-06-28T22:19:49.474ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37784
17272024-06-28T22:19:49.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45415 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37784 req_id = 1c37ff12-820e-483a-9260-e395afdc864f response_code = 204 uri = /switch-ports
17282024-06-28T22:19:49.522ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37784 req_id = 4af95bc2-08b0-4dfb-b9f6-db566eb42cc4 response_code = 200 uri = /network-bootstore-config
17292024-06-28T22:19:51.049ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 561 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = 82809bd5-47ea-46e3-acaa-6f85c81de6c8 response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67
17302024-06-28T22:20:16.663ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51755
17312024-06-28T22:20:16.663ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 77 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51755 req_id = c06d561c-d559-4d34-9bbd-ef61c0d71519 response_code = 200 uri = /baseboard
17322024-06-28T22:20:19.562ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63667
17332024-06-28T22:20:19.562ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63667 req_id = 9e3faf71-6162-40bb-91e3-fca08e266609 response_code = 200 uri = /v2p
17342024-06-28T22:20:19.570ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35858
17352024-06-28T22:20:19.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 830 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35858 req_id = b20f49cd-ab64-4877-bff5-d5a97742146c response_code = 200 uri = /vpc-routes
17362024-06-28T22:20:19.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35858 req_id = 7837fe1b-4a25-46a7-951d-3b8ff79349f8 response_code = 204 uri = /vpc-routes
17372024-06-28T22:20:19.580ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60883
17382024-06-28T22:20:19.580ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:690 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5ab9e19a-f9b8-42cb-b346-aff1c689c92e, kind: Service { id: a52585f4-7208-41c3-ac37-3029cbd9b012 }, name: Name("external-dns-a52585f4-7208-41c3-ac37-3029cbd9b012"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: a20b9ed9-42a5-4070-ba7b-11203172977a, kind: Service { id: effcf430-6675-455f-820f-495829fc7071 }, name: Name("external-dns-effcf430-6675-455f-820f-495829fc7071"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 155])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 35808902-45c0-47d5-8544-cb7fdaca7e8c, kind: Service { id: 88974d37-08ca-4371-bfcc-d8e58336bbf5 }, name: Name("nexus-88974d37-08ca-4371-bfcc-d8e58336bbf5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 69])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ba613ac1-2766-48e3-acdb-eb29ca6e137f, kind: Service { id: 09df608b-9da6-413f-9a4b-1808bcd7a800 }, name: Name("nexus-09df608b-9da6-413f-9a4b-1808bcd7a800"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 109])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f475e35c-414a-4bcc-b0ee-9464f4e4dbaf, kind: Service { id: 221b6ce2-61f2-4949-9537-b4af13ae839b }, name: Name("nexus-221b6ce2-61f2-4949-9537-b4af13ae839b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 119])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17392024-06-28T22:20:19.580ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte0" rules = []
17402024-06-28T22:20:19.580ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17412024-06-28T22:20:19.580ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17422024-06-28T22:20:19.580ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17432024-06-28T22:20:19.580ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17442024-06-28T22:20:19.580ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:709 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17452024-06-28T22:20:19.580ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60883 req_id = 8e8d0eab-0086-4c29-89b4-0d4d7c85371f response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17462024-06-28T22:20:19.691ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33102
17472024-06-28T22:20:19.692ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17482024-06-28T22:20:19.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 84289 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33102 req_id = ac87aabe-e1f4-441c-9b33-b63086c0e9d5 response_code = 200 uri = /inventory
17492024-06-28T22:20:19.776ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17502024-06-28T22:20:19.776ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/8cbaafb4-c085-4997-8977-45726fe29b62/config/omicron-zones.json file = common/src/ledger.rs:177
17512024-06-28T22:20:19.777ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/aac5d29e-4a80-46cd-84a5-ef3e32cd78c3/config/omicron-zones.json file = common/src/ledger.rs:177
17522024-06-28T22:20:19.777ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32859
17532024-06-28T22:20:19.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1062 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33102 req_id = ba1c3110-adb3-4bd5-85ed-114bad04a3b1 response_code = 200 uri = /omicron-zones
17542024-06-28T22:20:19.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44122 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:32859 req_id = d67efbd7-b674-4f81-96cb-fc19e28d94b4 response_code = 204 uri = /switch-ports
17552024-06-28T22:20:19.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32859 req_id = 41743639-0e0d-4bba-b023-1d280477809b response_code = 200 uri = /network-bootstore-config
17562024-06-28T22:20:46.695ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43886
17572024-06-28T22:20:46.695ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 83 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43886 req_id = 93b76d1a-7a2d-4afa-8e68-ec26fbaff440 response_code = 200 uri = /baseboard
17582024-06-28T22:20:49.237ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32942
17592024-06-28T22:20:49.237ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32942 req_id = 66755180-ee26-47a6-bae3-e6a56f54314b response_code = 200 uri = /vpc-routes
17602024-06-28T22:20:49.242ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:32942 req_id = ef77ca5e-7e61-4f41-939a-f71786a7298c response_code = 204 uri = /vpc-routes
17612024-06-28T22:20:49.252ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48288
17622024-06-28T22:20:49.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 750 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48288 req_id = f54a1341-412f-46ef-b885-aca6df0f7c6d response_code = 200 uri = /v2p
17632024-06-28T22:20:49.438ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59993
17642024-06-28T22:20:49.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44075 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59993 req_id = c3b13ab4-4dee-430d-8351-69cb751e1336 response_code = 204 uri = /switch-ports
17652024-06-28T22:20:49.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59993 req_id = dbd89cb7-6388-4413-9e78-1bdb408742ba response_code = 200 uri = /network-bootstore-config
17662024-06-28T22:20:51.137ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1346 local_addr = [fd00:1122:3344:101::1]:41782 method = GET remote_addr = [fd00:1122:3344:101::d]:65345 req_id = e7af68f6-f74c-47eb-90bb-01b2daec8f2e response_code = 200 sled_id = a9021c63-7f96-431e-9198-9a7654ae0f67 uri = /a9021c63-7f96-431e-9198-9a7654ae0f67